MaskDiffusion: Boosting Text-to-Image Consistency with Conditional Mask
Recent advancements in diffusion models have showcased their impressive capacity to generate visually striking images. Nevertheless, ensuring a close match between the generated image and the given prompt remains a persistent challenge. In this work, we identify that a crucial factor leading to the text-image mismatch issue is the inadequate cross-modality relation learning between the prompt and the output image. To better align the prompt and image content, we advance the cross-attention with an adaptive mask, which is conditioned on the attention maps and the prompt embeddings, to dynamically adjust the contribution of each text token to the image features. This mechanism explicitly diminishes the ambiguity in semantic information embedding from the text encoder, leading to a boost of text-to-image consistency in the synthesized images. Our method, termed MaskDiffusion, is training-free and hot-pluggable for popular pre-trained diffusion models. When applied to the latent diffusion models, our MaskDiffusion can significantly improve the text-to-image consistency with negligible computation overhead compared to the original diffusion models.
Code (0)
등록된 구현이 없습니다.
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
MaskDiffusion: Exploiting Pre-trained Diffusion Models for Semantic Segmentation
Semantic segmentation is essential in computer vision for various applications, yet traditional approaches face significant challenges, including the high cost of annotation and extensive training for supervised learning…
Open Vocabulary Semantic SegmentationOpen-Vocabulary Semantic SegmentationProper NounSegmentation+1Entropy Rectifying Guidance for Diffusion and Flow Models
Guidance techniques are commonly used in diffusion and flow models to improve image quality and consistency for conditional generative tasks such as class-conditional and text-to-image generation. In particular, classifi…
DiversityImage GenerationText to Image GenerationText-to-Image Generation+1Boosting Consistency in Story Visualization with Rich-Contextual Conditional Diffusion Models
Recent research showcases the considerable potential of conditional diffusion models for generating consistent stories. However, current methods, which predominantly generate stories in an autoregressive and excessively …
Story VisualizationTowards High-Fidelity Text-Guided 3D Face Generation and Manipulation Using only Images
Generating 3D faces from textual descriptions has a multitude of applications, such as gaming, movie, and robotics. Recent progresses have demonstrated the success of unconditional 3D face generation and text-to-3D shape…
3D Shape GenerationContrastive Learningcross-modal alignmentFace Generation+2Conditional Consistency Regularization for Semi-Supervised Multi-label Image Classification
Consistency regularization has achieved great successes in Semi-Supervised Single-Label Image Classification (SS-SLC) with deep learning models, while few effort has been devoted to Semi-Supervised Multi-Label Image Clas…
Classificationimage-classificationImage ClassificationMulti-Label Image Classification