Positive-Augmented Contrastive Learning for Image and Video Captioning Evaluation
The CLIP model has been recently proven to be very effective for a variety of cross-modal tasks, including the evaluation of captions generated from vision-and-language architectures. In this paper, we propose a new recipe for a contrastive-based evaluation metric for image captioning, namely Positive-Augmented Contrastive learning Score (PAC-S), that in a novel way unifies the learning of a contrastive visual-semantic space with the addition of generated images and text on curated data. Experiments spanning several datasets demonstrate that our new metric achieves the highest correlation with human judgments on both images and videos, outperforming existing reference-based metrics like CIDEr and SPICE and reference-free metrics like CLIP-Score. Finally, we test the system-level correlation of the proposed metric when considering popular image captioning approaches, and assess the impact of employing different cross-modal features. Our source code and trained models are publicly available at: https://github.com/aimagelab/pacscore.
Code (1)
Tasks
Contrastive LearningImage CaptioningVideo CaptioningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
RCA-NOC: Relative Contrastive Alignment for Novel Object Captioning
In this paper, we introduce a novel approach to novel object captioning which employs relative contrastive learning to learn visual and semantic alignment. Our approach maximizes compatibility between regions and object …
Contrastive LearningObjectTAGPositive-Augmented Contrastive Learning for Vision-and-Language Evaluation and Training
Despite significant advancements in caption generation, existing evaluation metrics often fail to capture the full quality or fine-grained details of captions. This is mainly due to their reliance on non-specific human-w…
Caption GenerationContrastive LearningSupervised Contrastive Frame Aggregation for Video Representation Learning
We propose a supervised contrastive learning framework for video representation learning that leverages temporally global context. We introduce a video to image aggregation strategy that spatially arranges multiple frame…
Representation LearningContrastive LearningData AugmentationStyle-Aware Contrastive Learning for Multi-Style Image Captioning
Existing multi-style image captioning methods show promising results in generating a caption with accurate visual content and desired linguistic style. However, existing methods overlook the relationship between linguist…
Contrastive LearningImage CaptioningRetrievalTripletOW-VISCapTor: Abstractors for Open-World Video Instance Segmentation and Captioning
We propose the new task 'open-world video instance segmentation and captioning'. It requires to detect, segment, track and describe with rich captions never before seen objects. This challenging task can be addressed by …
DescriptiveDiversityInstance SegmentationLanguage Modeling+7