CLIP-Diffusion-LM: Apply Diffusion Model on Image Captioning
Image captioning task has been extensively researched by previous work. However, limited experiments focus on generating captions based on non-autoregressive text decoder. Inspired by the recent success of the denoising diffusion model on image synthesis tasks, we apply denoising diffusion probabilistic models to text generation in image captioning tasks. We show that our CLIP-Diffusion-LM is capable of generating image captions using significantly fewer inference steps than autoregressive models. On the Flickr8k dataset, the model achieves 0.1876 BLEU-4 score. By training on the combined Flickr8k and Flickr30k dataset, our model achieves 0.2470 BLEU-4 score. Our code is available at https://github.com/xu-shitong/diffusion-image-captioning.
Code (1)
Tasks
DecoderDenoisingImage CaptioningImage GenerationText GenerationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
UniBrain: Unify Image Reconstruction and Captioning All in One Diffusion Model from Human Brain Activity
Image reconstruction and captioning from brain activity evoked by visual stimuli allow researchers to further understand the connection between the human brain and the visual perception system. While deep generative mode…
AllBrain DecodingImage CaptioningImage ReconstructionDiffCap: Exploring Continuous Diffusion on Image Captioning
Current image captioning works usually focus on generating descriptions in an autoregressive manner. However, there are limited works that focus on generating descriptions non-autoregressively, which brings more decoding…
Caption GenerationDiversityImage CaptioningImage Description+2Diffusion Bridge: Leveraging Diffusion Model to Reduce the Modality Gap Between Text and Vision for Zero-Shot Image Captioning
The modality gap between vision and text embeddings in CLIP presents a significant challenge for zero-shot image captioning, limiting effective cross-modal representation. Traditional approaches, such as noise inject…
cross-modal alignmentDenoisingImage CaptioningPrefix-diffusion: A Lightweight Diffusion Model for Diverse Image Captioning
While impressive performance has been achieved in image captioning, the limited diversity of the generated captions and the large parameter scale remain major barriers to the real-word application of these systems. In th…
DenoisingDiversityImage Captioningclip2latent: Text driven sampling of a pre-trained StyleGAN using denoising diffusion and CLIP
We introduce a new method to efficiently create text-to-image models from a pre-trained CLIP and StyleGAN. It enables text driven sampling with an existing generative model without any external data or fine-tuning. This …
Denoising