Understanding Self-supervised Learning via Information Bottleneck Principle
Self-supervised learning alleviates the massive demands for annotations in deep learning, and recent advances are mainly dominated by contrastive learning. Existed contrastive learning methods narrows the distance between positive pair while neglect the redundancy shared by positive pairs. To address this issue, we introduce the information bottleneck principle and propose the Self-supervised Variational Information Bottleneck (SVIB) learning framework. Specifically, We apply Gaussian Mixture Model (GMM) as the parametric approximate posterior distribution to the real feature distribution and introduce the categorical latent variable. Features from different augmentations are forced to infer the other one and the latent variable together. Then, we propose variational information bottleneck as our objective, which is composed of two parts. The first is maximizing the mutual information between the inferred feature and the latent variable. The second is minimizing the mutual information between the other feature and the latent variable. Compare to previous works, SVIB provides the self-supervised learning field with a novel perspective from the variational information bottleneck, while also highlighting a long-neglected issue. Experiments show that SVIB outperforms current SOTA methods in multiple benchmarks.
Code (0)
등록된 구현이 없습니다.
Tasks
Contrastive LearningSelf-Supervised LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
BottleSum: Unsupervised and Self-supervised Sentence Summarization using the Information Bottleneck Principle
The principle of the Information Bottleneck (Tishby et al. 1999) is to produce a summary of information X optimized to predict some other relevant information Y. In this paper, we propose a novel approach to unsupervised…
Abstractive Text SummarizationExtractive SummarizationLanguage ModelingLanguage Modelling+4Self-Supervised Information Bottleneck for Deep Multi-View Subspace Clustering
In this paper, we explore the problem of deep multi-view subspace clustering framework from an information-theoretic point of view. We extend the traditional information bottleneck principle to learn common information a…
ClusteringMulti-view Subspace ClusteringLearning Mask Invariant Mutual Information for Masked Image Modeling
Masked autoencoders (MAEs) represent a prominent self-supervised learning paradigm in computer vision. Despite their empirical success, the underlying mechanisms of MAEs remain insufficiently understood. Recent studies h…
Contrastive Learningimage-classificationImage Classificationobject-detection+3Why Self-Supervised Encoders Want to Be Normal
Self-supervised learning has achieved remarkable empirical success in learning robust representations without explicit labels, most recently demonstrated within the framework of Joint-Embedding Predictive Architectures (…
Self-Supervised LearningTo Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Review
Deep neural networks excel in supervised learning tasks but are constrained by the need for extensive labeled data. Self-supervised learning emerges as a promising alternative, allowing models to learn without explicit l…
Self-Supervised Learning