RealignDiff: Boosting Text-to-Image Diffusion Model with Coarse-to-fine Semantic Re-alignment
Recent advances in text-to-image diffusion models have achieved remarkable success in generating high-quality, realistic images from textual descriptions. However, these approaches have faced challenges in precisely aligning the generated visual content with the textual concepts described in the prompts. In this paper, we propose a two-stage coarse-to-fine semantic re-alignment method, named RealignDiff, aimed at improving the alignment between text and images in text-to-image diffusion models. In the coarse semantic re-alignment phase, a novel caption reward, leveraging the BLIP-2 model, is proposed to evaluate the semantic discrepancy between the generated image caption and the given text prompt. Subsequently, the fine semantic re-alignment stage employs a local dense caption generation module and a re-weighting attention modulation module to refine the previously generated images from a local semantic view. Experimental results on the MS-COCO and ViLG-300 datasets demonstrate that the proposed two-stage coarse-to-fine semantic re-alignment method outperforms other baseline re-alignment techniques by a substantial margin in both visual quality and semantic similarity with the input prompt.
Code (0)
등록된 구현이 없습니다.
Tasks
Caption GenerationLanguage ModellingLarge Language ModelSemantic SimilaritySemantic Textual SimilarityMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Sherpa3D: Boosting High-Fidelity Text-to-3D Generation via Coarse 3D Prior
Recently, 3D content creation from text prompts has demonstrated remarkable progress by utilizing 2D and 3D diffusion models. While 3D diffusion models ensure great multi-view consistency, their ability to generate high-…
3D GenerationText to 3DDreamix: Video Diffusion Models are General Video Editors
Text-driven image and video diffusion models have recently achieved unprecedented generation realism. While diffusion models have been successfully applied for image editing, very few works have done so for video editing…
Image AnimationImage to Video GenerationSubject-driven Video GenerationText-to-Video Editing+2Boosting Diffusion Model for Spectrogram Up-sampling in Text-to-speech: An Empirical Study
Scaling text-to-speech (TTS) with autoregressive language model (LM) to large-scale datasets by quantizing waveform into discrete speech tokens is making great progress to capture the diversity and expressiveness in huma…
DiversityLanguage ModelingLanguage Modellingtext-to-speech+1Inversion Circle Interpolation: Diffusion-based Image Augmentation for Data-scarce Classification
Data Augmentation (DA), i.e., synthesizing faithful and diverse samples to expand the original training set, is a prevalent and effective strategy to improve the performance of various data-scarce tasks. With the powerfu…
ClassificationData AugmentationDenoisingDiversity+4Texture Generation on 3D Meshes with Point-UV Diffusion
In this work, we focus on synthesizing high-quality textures on 3D meshes. We present Point-UV diffusion, a coarse-to-fine pipeline that marries the denoising diffusion model with UV mapping to generate 3D consistent and…
DenoisingTexture Synthesis