Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
Cross-modal contrastive learning in vision language pretraining (VLP) faces the challenge of (partial) false negatives. In this paper, we study this problem from the perspective of Mutual Information (MI) optimization. It is common sense that InfoNCE loss used in contrastive learning will maximize the lower bound of MI between anchors and their positives, while we theoretically prove that MI involving negatives also matters when noises commonly exist. Guided by a more general lower bound form for optimization, we propose a contrastive learning strategy regulated by progressively refined cross-modal similarity, to more accurately optimize MI between an image/text anchor and its negative texts/images instead of improperly minimizing it. Our method performs competitively on four downstream cross-modal tasks and systematically balances the beneficial and harmful effects of (partial) false negative samples under theoretical guidance.
Code (0)
등록된 구현이 없습니다.
Tasks
Common Sense ReasoningContrastive LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
CoCo-BERT: Improving Video-Language Pre-training with Contrastive Cross-modal Matching and Denoising
BERT-type structure has led to the revolution of vision-language pre-training and the achievement of state-of-the-art results on numerous vision-language downstream tasks. Existing solutions dominantly capitalize on the …
Cross-Modal RetrievalDecoderDenoisingLanguage Modeling+6LeVLJEPA: End-to-End Vision-Language Pretraining Without Negatives
Vision-language pretraining remains dominated by contrastive objectives, whereas vision-only self-supervised learning has largely adopted non-contrastive methods. At the same time, the role of vision-language encoders ha…
Self-Supervised LearningSemantic SegmentationOn the Language Encoder of Contrastive Cross-modal Models
Contrastive cross-modal models such as CLIP and CLAP aid various vision-language (VL) and audio-language (AL) tasks. However, there has been limited investigation of and improvement in their language encoder, which is th…
cross-modal alignmentSentenceSentence EmbeddingSentence-EmbeddingVision-Language Pre-Training with Triple Contrastive Learning
Vision-language representation learning largely benefits from image-text alignment through contrastive losses (e.g., InfoNCE loss). The success of this alignment strategy is attributed to its capability in maximizing the…
Contrastive Learningcross-modal alignmentCross-Modal RetrievalImage-text Retrieval+7COOKIE: Contrastive Cross-Modal Knowledge Sharing Pre-Training for Vision-Language Representation
There has been a recent surge of interest in cross-modal pre-training. However, existed approaches pre-train a one-stream model to learn joint vision-language representation, which suffers from calculation explosion …
Contrastive LearningCross-Modal RetrievalImage RetrievalRetrieval+1