Dense Text-to-Image Generation with Attention Modulation
Existing text-to-image diffusion models struggle to synthesize realistic images given dense captions, where each text prompt provides a detailed description for a specific image region. To address this, we propose DenseDiffusion, a training-free method that adapts a pre-trained text-to-image model to handle such dense captions while offering control over the scene layout. We first analyze the relationship between generated images' layouts and the pre-trained model's intermediate attention maps. Next, we develop an attention modulation method that guides objects to appear in specific regions according to layout guidance. Without requiring additional fine-tuning or datasets, we improve image generation performance given dense captions regarding both automatic and human evaluation scores. In addition, we achieve similar-quality visual results with models specifically trained with layout conditions.
Code (1)
Tasks
Image GenerationText to Image GenerationText-to-Image GenerationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
On-the-Fly Attention Modulation for Neural Generation
Despite considerable advancements with deep neural language models (LMs), neural text generation still suffers from degeneration: the generated text is repetitive, generic, self-contradictory, and often lacks commonsense…
Language ModellingSentenceText GenerationSelf-supervised Learning with Speech Modulation Dropout
We show that training a multi-headed self-attention-based deep network to predict deleted, information-dense 2-8 Hz speech modulations over a 1.5-second section of a speech utterance is an effective way to make machines …
Automatic Speech RecognitionSelf-Supervised Learningspeech-recognitionSpeech RecognitionRethinking Global Text Conditioning in Diffusion Transformers
Diffusion transformers typically incorporate textual information via attention layers and a modulation mechanism using a pooled text embedding. Nevertheless, recent approaches discard modulation-based text conditioning a…
Video GenerationImage EditingTokenVerse: Versatile Multi-concept Personalization in Token Modulation Space
We present TokenVerse -- a method for multi-concept personalization, leveraging a pre-trained text-to-image diffusion model. Our framework can disentangle complex visual elements and attributes from as little as a single…
Lightweight Image Enhancement Network for Mobile Devices Using Self-Feature Extraction and Dense Modulation
Convolutional neural network (CNN) based image enhancement methods such as super-resolution and detail enhancement have achieved remarkable performances. However, amounts of operations including convolution and parameter…
Image EnhancementSuper-Resolution