MVEB: Self-Supervised Learning with Multi-View Entropy Bottleneck

2024-03-28 · Liangjian Wen, Xiasi Wang, Jianzhuang Liu, Zenglin Xu

Self-supervised learning aims to learn representation that can be effectively generalized to downstream tasks. Many self-supervised approaches regard two views of an image as both the input and the self-supervised signals, assuming that either view contains the same task-relevant information and the shared information is (approximately) sufficient for predicting downstream tasks. Recent studies show that discarding superfluous information not shared between the views can improve generalization. Hence, the ideal representation is sufficient for downstream tasks and contains minimal superfluous information, termed minimal sufficient representation. One can learn this representation by maximizing the mutual information between the representation and the supervised view while eliminating superfluous information. Nevertheless, the computation of mutual information is notoriously intractable. In this work, we propose an objective termed multi-view entropy bottleneck (MVEB) to learn minimal sufficient representation effectively. MVEB simplifies the minimal sufficient learning to maximizing both the agreement between the embeddings of two views and the differential entropy of the embedding distribution. Our experiments confirm that MVEB significantly improves performance. For example, it achieves top-1 accuracy of 76.9\% on ImageNet with a vanilla ResNet-50 backbone on linear evaluation. To the best of our knowledge, this is the new state-of-the-art result with ResNet-50.

📄 PDF Abstract BibTeX arXiv:2403.19078

Code (1)

wlj6816/MVEB pytorch

Tasks

Linear evaluationSelf-Supervised Learning

MVEB: Massive Video Embedding Benchmark

2026-06-12 · Adnan El Assadi, Roman Solomatin, Isaac Chung, Chenghao Xiao 외 arxiv

We introduce the Massive Video Embedding Benchmark (MVEB), a 23-task benchmark for video embeddings spanning classification, zero-shot classification, clustering, pair classification, retrieval, and video-centric questio…

Question Answering

The Role of Entropy and Reconstruction in Multi-View Self-Supervised Learning

2023-07-20 · Borja Rodríguez-Gálvez, Arno Blaas, Pau Rodríguez, Adam Goliński 외

The mechanisms behind the success of multi-view self-supervised learning (MVSSL) are not yet fully understood. Contrastive MVSSL methods have been studied through the lens of InfoNCE, a lower bound of the Mutual Informat…

Self-Supervised Learning

Self-Contrastive Learning

2021-09-29 · Sangmin Bae, Sungnyun Kim, Jongwoo Ko, Gihun Lee 외

This paper proposes a novel contrastive learning framework, called Self-Contrastive (SelfCon) Learning, that self-contrasts within multiple outputs from the different levels of a multi-exit network. SelfCon learning does…

Contrastive Learning

SimMER: Simple Maximization of Entropy and Rank for Self-supervised Representation Learning

2021-09-29 · Zhengyu Yang, Zijian Hu, Xuefeng Hu, Ram Nevatia

Consistency regularization, referring to enforcing consistency across a model's responses to different views of the same input, is widely used for self-supervised image representation learning. However, consistency regul…

Contrastive Learning Linear evaluation Representation Learning Self-Supervised Learning

Self-Supervised Representation Learning via Hyperspherical Density Shaping

2026-04-27 · Esteban Rodríguez-Betancourt, Edgar Casasola-Murillo arxiv

Modern self-supervised representation learning methods often relies on empirical heuristics that are not theoretically grounded. In this study we propose HyDeS, a theoretically grounded method based on multi-view mutual …

Self-Supervised Learning Representation Learning