Diffusion Lens: Interpreting Text Encoders in Text-to-Image Pipelines
Text-to-image diffusion models (T2I) use a latent representation of a text prompt to guide the image generation process. However, the process by which the encoder produces the text representation is unknown. We propose the Diffusion Lens, a method for analyzing the text encoder of T2I models by generating images from its intermediate representations. Using the Diffusion Lens, we perform an extensive analysis of two recent T2I models. Exploring compound prompts, we find that complex scenes describing multiple objects are composed progressively and more slowly compared to simple scenes; Exploring knowledge retrieval, we find that representation of uncommon concepts requires further computation compared to common concepts, and that knowledge retrieval is gradual across layers. Overall, our findings provide valuable insights into the text encoder component in T2I pipelines.
Code (0)
등록된 구현이 없습니다.
Tasks
Image GenerationRetrievalMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Decoding Vision Transformers: the Diffusion Steering Lens
Logit Lens is a widely adopted method for mechanistic interpretability of transformer-based language models, enabling the analysis of how internal representations evolve across layers by projecting them into the output v…
Interpreting Large Text-to-Image Diffusion Models with Dictionary Learning
Sparse autoencoders are a promising new approach for decomposing language model activations for interpretation and control. They have been applied successfully to vision transformer image encoders and to small-scale diff…
Dictionary LearningImage GenerationLanguage ModelingLanguage ModellingInterpreting and Analysing CLIP's Zero-Shot Image Classification via Mutual Knowledge
Contrastive Language-Image Pretraining (CLIP) performs zero-shot image classification by mapping images and textual class representation into a shared embedding space, then retrieving the class closest to the image. This…
Classificationimage-classificationImage Classificationzero-shot-classification+2Schödinger Bridge Type Diffusion Models as an Extension of Variational Autoencoders
Generative diffusion models use time-forward and backward stochastic differential equations to connect the data and prior distributions. While conventional diffusion models (e.g., score-based models) only learn the backw…
$\textit{Revelio}$: Interpreting and leveraging semantic information in diffusion models
We study $\textit{how}$ rich visual semantic information is represented within various layers and denoising timesteps of different diffusion architectures. We uncover monosemantic interpretable features by leveraging k-s…
DenoisingLanguage ModelingLanguage ModellingRepresentation Learning+1