CLUB: A Contrastive Log-ratio Upper Bound of Mutual Information
Mutual information (MI) minimization has gained considerable interests in various machine learning tasks. However, estimating and minimizing MI in high-dimensional spaces remains a challenging problem, especially when only samples, rather than distribution forms, are accessible. Previous works mainly focus on MI lower bound approximation, which is not applicable to MI minimization problems. In this paper, we propose a novel Contrastive Log-ratio Upper Bound (CLUB) of mutual information. We provide a theoretical analysis of the properties of CLUB and its variational approximation. Based on this upper bound, we introduce a MI minimization training scheme and further accelerate it with a negative sampling strategy. Simulation studies on Gaussian distributions show the reliable estimation ability of CLUB. Real-world MI minimization experiments, including domain adaptation and information bottleneck, demonstrate the effectiveness of the proposed method. The code is at https://github.com/Linear95/CLUB.
Code (2)
Tasks
Domain AdaptationSimilar Papers 제목 키워드 기반
Disentangled Speaker Representation Learning via Mutual Information Minimization
Domain mismatch problem caused by speaker-unrelated feature has been a major topic in speaker recognition. In this paper, we propose an explicit disentanglement framework to unravel speaker-relevant features from speaker…
DisentanglementRepresentation LearningSpeaker RecognitionSpeaker Verification+1Mutual Information Estimation as a Difference of Entropies for Unsupervised Representation Learning
Contrastive loss has been successfully exploited in the latest visual unsupervised representation learning methods. Contrastive loss is based on a lower-bound estimation of mutual information where its known limitations …
Mutual Information EstimationRepresentation LearningJoint Contrastive Learning for Unsupervised Domain Adaptation
Enhancing feature transferability by matching marginal distributions has led to improvements in domain adaptation, although this is at the expense of feature discrimination. In particular, the ideal joint hypothesis erro…
Contrastive LearningDomain AdaptationUnsupervised Domain AdaptationMIO : Mutual Information Optimization using Self-Supervised Binary Contrastive Learning
Self-supervised contrastive learning frameworks have progressed rapidly over the last few years. In this paper, we propose a novel loss function for contrastive learning. We model our pre-training task as a binary classi…
Binary ClassificationContrastive LearningLinear evaluationTowards Generalizable Graph Contrastive Learning: An Information Theory Perspective
Graph contrastive learning (GCL) emerges as the most representative approach for graph representation learning, which leverages the principle of maximizing mutual information (InfoMax) to learn node representations appli…
Contrastive LearningData AugmentationGraph Representation LearningRepresentation Learning