Understanding Self-supervised Contrastive Learning through Supervised Objectives
Self-supervised representation learning has achieved impressive empirical success, yet its theoretical understanding remains limited. In this work, we provide a theoretical perspective by formulating self-supervised representation learning as an approximation to supervised representation learning objectives. Based on this formulation, we derive a loss function closely related to popular contrastive losses such as InfoNCE, offering insight into their underlying principles. Our derivation naturally introduces the concepts of prototype representation bias and a balanced contrastive loss, which help explain and improve the behavior of self-supervised learning algorithms. We further show how components of our theoretical framework correspond to established practices in contrastive learning. Finally, we empirically validate the effect of balancing positive and negative pair interactions. All theoretical proofs are provided in the appendix, and our code is included in the supplementary material.
Code (0)
등록된 구현이 없습니다.
Tasks
Self-Supervised LearningRepresentation LearningContrastive LearningSimilar Papers 제목 키워드 기반
Understanding Self-Supervised Pretraining with Part-Aware Representation Learning
In this paper, we are interested in understanding self-supervised pretraining through studying the capability that self-supervised representation pretraining methods learn part-aware representations. The study is mainly …
Contrastive LearningObjectRepresentation LearningCrossVideo: Self-supervised Cross-modal Contrastive Learning for Point Cloud Video Understanding
This paper introduces a novel approach named CrossVideo, which aims to enhance self-supervised cross-modal contrastive learning in the field of point cloud video understanding. Traditional supervised learning methods enc…
Contrastive Learningpoint cloud video understandingSelf-Supervised LearningVideo UnderstandingVariational Self-Supervised Contrastive Learning Using Beta Divergence
Learning a discriminative semantic space using unlabelled and noisy data remains unaddressed in a multi-label setting. We present a contrastive self-supervised learning method which is robust to data noise, grounded in t…
Face RecognitionLinear evaluationSelf-Supervised LearningUnderstanding The Robustness of Self-supervised Learning Through Topic Modeling
Self-supervised learning has significantly improved the performance of many NLP tasks. However, how can self-supervised learning discover useful representations, and why is it better than traditional approaches such as p…
Self-Supervised LearningTopic ModelsRePre: Improving Self-Supervised Vision Transformer with Reconstructive Pre-training
Recently, self-supervised vision transformers have attracted unprecedented attention for their impressive representation learning ability. However, the dominant method, contrastive learning, mainly relies on an instance …
Contrastive LearningDecoderRepresentation Learning