On the modality gap and the contrastive loss in multi-modal representation learning
We study the modality gap in CLIP-style dual-encoder contrastive learning, where image and text embeddings remain misaligned despite being trained in a shared space. We argue that the gap is induced by a failure of the InfoNCE formulation with independent encoders. We conduct a uni-modal experiment with two independent encoders and identical initialization conditions and find that InfoNCE actively generates a gap at low temperatures. We provide a theoretical analysis of this phenomenon and show that the modality gap is indeed a mode-failure of InfoNCE, but only at low temperatures. We propose a simple modification called xNCE, which uses intermodal as well as intra-modality negative contrastive pairs. xNCE matches retrieval performance on MS-COCO while consistently reducing the gap even at low temperatures. Notably, xNCE improves zero-shot classification over the InfoNCE baseline across all benchmarks, whereas high-temperature InfoNCE and regularized InfoNCE both fail to do so, demonstrating that xNCE reduces the modality gap without sacrificing the discriminative geometry needed for transfer.
Code (0)
등록된 구현이 없습니다.
Tasks
Representation LearningContrastive LearningSimilar Papers 제목 키워드 기반
It's Not a Modality Gap: Characterizing and Addressing the Contrastive Gap
Multi-modal contrastive models such as CLIP achieve state-of-the-art performance in zero-shot classification by embedding input images and texts on a joint representational space. Recently, a modality gap has been report…
image-classificationImage Classificationzero-shot-classificationZero-Shot Image Classification+1Understanding and Constructing Latent Modality Structures in Multi-modal Representation Learning
Contrastive loss has been increasingly used in learning representations from multiple modalities. In the limit, the nature of the contrastive loss encourages modalities to exactly match each other in the latent space. Ye…
Few-Shot Image Classificationimage-classificationImage ClassificationImage-text Retrieval+9Combining Reconstruction and Contrastive Methods for Multimodal Representations in RL
Learning self-supervised representations using reconstruction or contrastive losses improves performance and sample complexity of image-based and multimodal reinforcement learning (RL). Here, different self-supervised lo…
reinforcement-learningReinforcement Learning (RL)Representation LearningMutual Contrastive Low-rank Learning to Disentangle Whole Slide Image Representations for Glioma Grading
Whole slide images (WSI) provide valuable phenotypic information for histological assessment and malignancy grading of tumors. The WSI-based grading promises to provide rapid diagnostic support and facilitate digital hea…
Contrastive LearningDiagnosticDisentanglementwhole slide imagesMind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning
We present modality gap, an intriguing geometric phenomenon of the representation space of multi-modal models. Specifically, we show that different data modalities (e.g. images and text) are embedded at arm's length in t…
Contrastive LearningFairnessRepresentation Learningzero-shot-classification+1