Contrastive Learning for Image Captioning
Image captioning, a popular topic in computer vision, has achieved substantial progress in recent years. However, the distinctiveness of natural descriptions is often overlooked in previous work. It is closely related to the quality of captions, as distinctive captions are more likely to describe images with their unique aspects. In this work, we propose a new learning method, Contrastive Learning (CL), for image captioning. Specifically, via two constraints formulated on top of a reference model, the proposed method can encourage distinctiveness, while maintaining the overall quality of the generated captions. We tested our method on two challenging datasets, where it improves the baseline model by significant margins. We also showed in our studies that the proposed method is generic and can be used for models with various structures.
Code (0)
등록된 구현이 없습니다.
Tasks
Contrastive LearningImage CaptioningSimilar Papers 제목 키워드 기반
Positive-Augmented Contrastive Learning for Image and Video Captioning Evaluation
The CLIP model has been recently proven to be very effective for a variety of cross-modal tasks, including the evaluation of captions generated from vision-and-language architectures. In this paper, we propose a new reci…
Contrastive LearningImage CaptioningVideo CaptioningStyle-Aware Contrastive Learning for Multi-Style Image Captioning
Existing multi-style image captioning methods show promising results in generating a caption with accurate visual content and desired linguistic style. However, existing methods overlook the relationship between linguist…
Contrastive LearningImage CaptioningRetrievalTripletImage Captioners Are Scalable Vision Learners Too
Contrastive pretraining on image-text pairs from the web is one of the most popular large-scale pretraining strategies for vision backbones, especially in the context of large multimodal models. At the same time, image c…
DecoderImage CaptioningRCA-NOC: Relative Contrastive Alignment for Novel Object Captioning
In this paper, we introduce a novel approach to novel object captioning which employs relative contrastive learning to learn visual and semantic alignment. Our approach maximizes compatibility between regions and object …
Contrastive LearningObjectTAGContext-Aware Group Captioning via Self-Attention and Contrastive Features
While image captioning has progressed rapidly, existing works focus mainly on describing single images. In this paper, we introduce a new task, context-aware group captioning, which aims to describe a group of target ima…
Image Captioning