Prefix-diffusion: A Lightweight Diffusion Model for Diverse Image Captioning
While impressive performance has been achieved in image captioning, the limited diversity of the generated captions and the large parameter scale remain major barriers to the real-word application of these systems. In this work, we propose a lightweight image captioning network in combination with continuous diffusion, called Prefix-diffusion. To achieve diversity, we design an efficient method that injects prefix image embeddings into the denoising process of the diffusion model. In order to reduce trainable parameters, we employ a pre-trained model to extract image features and further design an extra mapping network. Prefix-diffusion is able to generate diverse captions with relatively less parameters, while maintaining the fluency and relevance of the captions benefiting from the generative capabilities of the diffusion model. Our work paves the way for scaling up diffusion models for image captioning, and achieves promising performance compared with recent approaches.
Code (0)
등록된 구현이 없습니다.
Tasks
DenoisingDiversityImage CaptioningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
TAPS: Target-Aware Prefix Tree Selection for Diffusion-Drafted Speculative Decoding
Using a diffusion model for parallel drafting is a promising approach for speculative decoding. By predicting tokens at multiple future positions in a single forward pass, diffusion drafters substantially reduce drafting…
Transfer Learning for Text Diffusion Models
In this report, we explore the potential for text diffusion to replace autoregressive (AR) decoding for the training and deployment of large language models (LLMs). We are particularly interested to see whether pretraine…
DecoderMachine TranslationText GenerationTransfer LearningD^2SD: Accelerating Speculative Decoding with Dual Diffusion Draft Models
Speculative decoding accelerates autoregressive large language model inference by drafting multiple tokens and verifying them in a single target-model forward pass. Recent diffusion-based drafters generate an entire bloc…
Prefix-Adaptive Block Diffusion for Efficient Document Recognition
Block Diffusion Models (BDMs) support parallel generation, flexible-length output, and KV caching, making them promising for efficient document parsing. However, existing BDMs bind denoising and cache commitment to fixed…
Anonymization Prompt Learning for Facial Privacy-Preserving Text-to-Image Generation
Text-to-image diffusion models, such as Stable Diffusion, generate highly realistic images from text descriptions. However, the generation of certain content at such high quality raises concerns. A prominent issue is the…
Face SwappingImage GenerationPrivacy PreservingPrompt Learning+2