Semi-Supervised Contrastive Learning with Orthonormal Prototypes
Contrastive learning has emerged as a powerful method in deep learning, excelling at learning effective representations through contrasting samples from different distributions. However, dimensional collapse, where embeddings converge into a lower-dimensional space, poses a significant challenge, especially in semi-supervised and self-supervised setups. In this paper, we first identify a critical learning-rate threshold, beyond which standard contrastive losses converge to collapsed solutions. Building on these insights, we propose CLOP, a novel semi-supervised loss function designed to prevent dimensional collapse by promoting the formation of orthogonal linear subspaces among class embeddings. Through extensive experiments on real and synthetic datasets, we demonstrate that CLOP improves performance in image classification and object detection tasks while also exhibiting greater stability across different learning rates and batch sizes.
Code (0)
등록된 구현이 없습니다.
Tasks
Contrastive LearningImage ClassificationObject DetectionSimilar Papers 제목 키워드 기반
Preventing Collapse in Contrastive Learning with Orthonormal Prototypes (CLOP)
Contrastive learning has emerged as a powerful method in deep learning, excelling at learning effective representations through contrasting samples from different distributions. However, neural collapse, where embeddings…
Contrastive LearningContraCluster: Learning to Classify without Labels by Contrastive Self-Supervision and Prototype-Based Semi-Supervision
The recent advances in representation learning inspire us to take on the challenging problem of unsupervised image classification tasks in a principled way. We propose ContraCluster, an unsupervised image classification …
Contrastive Learningimage-classificationImage ClassificationRepresentation Learning+2Open-world Semi-supervised Novel Class Discovery
Traditional semi-supervised learning tasks assume that both labeled and unlabeled data follow the same class distribution, but the realistic open-world scenarios are of more complexity with unknown novel classes mixed in…
Contrastive LearningNovel Class DiscoveryRepresentation LearningRepresentation Learning via Consistent Assignment of Views to Clusters
We introduce Consistent Assignment for Representation Learning (CARL), an unsupervised learning method to learn visual representations by combining ideas from self-supervised contrastive learning and deep clustering. By …
ClusteringContrastive LearningDeep ClusteringLinear evaluation+2Space Engage: Collaborative Space Supervision for Contrastive-based Semi-Supervised Semantic Segmentation
Semi-Supervised Semantic Segmentation (S4) aims to train a segmentation model with limited labeled images and a substantial volume of unlabeled images. To improve the robustness of representations, powerful methods intro…
Contrastive LearningSemantic SegmentationSemi-Supervised Semantic Segmentation