$f$-Mutual Information Contrastive Learning
Self-supervised contrastive learning is an emerging field due to its power in providing good data representations. Such learning paradigm widely adopts the InfoNCE loss, which is closely connected with maximizing the mutual information. In this work, we propose the $f$-Mutual Information Contrastive Learning framework ($f$-MICL) , which directly maximizes the $f$-divergence-based generalization of mutual information. We theoretically prove that, under mild assumptions, our $f$-MICL naturally attains the alignment for positive pairs and the uniformity for data representations, the two main factors for the success of contrastive learning. We further provide theoretical guidance on designing the similarity function and choosing the effective $f$-divergences for $f$-MICL. Using several benchmark tasks from both vision and natural text, we empirically verify that our novel method outperforms or performs on par with state-of-the-art strategies.
Code (0)
등록된 구현이 없습니다.
Tasks
Contrastive LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Towards a Rigorous Analysis of Mutual Information in Contrastive Learning
Contrastive learning has emerged as a cornerstone in recent achievements of unsupervised representation learning. Its primary paradigm involves an instance discrimination task with a mutual information loss. The loss is …
Contrastive LearningMisconceptionsRepresentation LearningRepresentation Learning via Non-Contrastive Mutual Information
Labeling data is often very time consuming and expensive, leaving us with a majority of unlabeled data. Self-supervised representation learning methods such as SimCLR (Chen et al., 2020) or BYOL (Grill et al., 2020) have…
Representation LearningMutual Contrastive Learning for Visual Representation Learning
We present a collaborative learning method called Mutual Contrastive Learning (MCL) for general visual representation learning. The core idea of MCL is to perform mutual interaction and transfer of contrastive distributi…
Contrastive LearningFew-Shot Learningimage-classificationImage Classification+6A Mutual Information Perspective on Federated Contrastive Learning
We investigate contrastive learning in the federated setting through the lens of SimCLR and multi-view mutual information maximization. In doing so, we uncover a connection between contrastive representation learning and…
Contrastive LearningFederated Unsupervised LearningRepresentation LearningCross-Domain Sentiment Classification with Contrastive Learning and Mutual Information Maximization
Contrastive learning (CL) has been successful as a powerful representation learning method. In this work we propose CLIM: Contrastive Learning with mutual Information Maximization, to explore the potential of CL on cross…
Contrastive LearningGeneral ClassificationRepresentation LearningSentiment Analysis+1