Latent Diffusion U-Net Representations Contain Positional Embeddings and Anomalies
Diffusion models have demonstrated remarkable capabilities in synthesizing realistic images, spurring interest in using their representations for various downstream tasks. To better understand the robustness of these representations, we analyze popular Stable Diffusion models using representational similarity and norms. Our findings reveal three phenomena: (1) the presence of a learned positional embedding in intermediate representations, (2) high-similarity corner artifacts, and (3) anomalous high-norm artifacts. These findings underscore the need to further investigate the properties of diffusion model representations before considering them for downstream tasks that require robust features. Project page: https://jonasloos.github.io/sd-representation-anomalies
Code (1)
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
GLASS: Guided Latent Slot Diffusion for Object-Centric Learning
Object-centric learning aims to decompose an input image into a set of meaningful object files (slots). These latent object representations enable a variety of downstream tasks. Yet, object-centric learning struggles…
Conditional Image GenerationImage GenerationObjectObject DiscoveryCompositional Discrete Latent Code for High Fidelity, Productive Diffusion Models
We argue that diffusion models' success in modeling complex distributions is, for the most part, coming from their input conditioning. This paper investigates the representation used to condition diffusion models from th…
Self-Supervised LearningText-to-Image GenerationDDMI: Domain-Agnostic Latent Diffusion Models for Synthesizing High-Quality Implicit Neural Representations
Recent studies have introduced a new class of generative models for synthesizing implicit neural representations (INRs) that capture arbitrary continuous signals in various domains. These models opened the door for domai…
3D Shape GenerationImage GenerationVideo GenerationEnergyMoGen: Compositional Human Motion Generation with Energy-Based Diffusion Model in Latent Space
Diffusion models, particularly latent diffusion models, have demonstrated remarkable success in text-driven human motion generation. However, it remains challenging for latent diffusion models to effectively compose mult…
Motion GenerationSemantic CompositionNot Only Text: Exploring Compositionality of Visual Representations in Vision-Language Models
Vision-Language Models (VLMs) learn a shared feature space for text and images, enabling the comparison of inputs of different modalities. While prior works demonstrated that VLMs organize natural language representation…