Generalizing Supervised Contrastive learning: A Projection Perspective
Self-supervised contrastive learning (SSCL) has emerged as a powerful paradigm for representation learning and has been studied from multiple perspectives, including mutual information and geometric viewpoints. However, supervised contrastive (SupCon) approaches have received comparatively little attention in this context: for instance, while InfoNCE used in SSCL is known to form a lower bound on mutual information (MI), the relationship between SupCon and MI remains unexplored. To address this gap, we introduce ProjNCE, a generalization of the InfoNCE loss that unifies supervised and self-supervised contrastive objectives by incorporating projection functions and an adjustment term for negative pairs. We prove that ProjNCE constitutes a valid MI bound and affords greater flexibility in selecting projection strategies for class embeddings. Building on this flexibility, we further explore the centroid-based class embeddings in SupCon by exploring a variety of projection methods. Extensive experiments on multiple datasets and settings demonstrate that ProjNCE consistently outperforms both SupCon and standard cross-entropy training. Our work thus refines SupCon along two complementary perspective--mutual information interpretation and projection design--and offers broadly applicable improvements whenever SupCon serves as the foundational contrastive objective.
Code (0)
등록된 구현이 없습니다.
Tasks
Contrastive LearningRepresentation LearningvalidMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Unified Framework for Feature Extraction based on Contrastive Learning
Feature extraction is an efficient approach for alleviating the issue of dimensionality in high-dimensional data. As a popular self-supervised learning method, contrastive learning has recently garnered considerable atte…
Contrastive LearningGraph EmbeddingSelf-Supervised LearningRefining Self-Supervised Learning in Imaging: Beyond Linear Metric
We introduce in this paper a new statistical perspective, exploiting the Jaccard similarity metric, as a measure-based metric to effectively invoke non-linear features in the loss of self-supervised contrastive learning.…
Contrastive LearningSelf-Supervised LearningTowards the Sparseness of Projection Head in Self-Supervised Learning
In recent years, self-supervised learning (SSL) has emerged as a promising approach for extracting valuable representations from unlabeled data. One successful SSL method is contrastive learning, which aims to bring posi…
Contrastive LearningSelf-Supervised LearningMulti-level Supervised Contrastive Learning
Contrastive learning is a well-established paradigm in representation learning. The standard framework of contrastive learning minimizes the distance between "similar" instances and maximizes the distance between dissimi…
Contrastive LearningRepresentation LearningUnderstanding and Improving the Role of Projection Head in Self-Supervised Learning
Self-supervised learning (SSL) aims to produce useful feature representations without access to any human-labeled data annotations. Due to the success of recent SSL methods based on contrastive learning, such as SimCLR, …
Contrastive Learningimage-classificationImage ClassificationSelf-Supervised Learning