Robust Contrastive Learning With Theory Guarantee
Contrastive learning (CL) is a self-supervised training paradigm that allows us to extract meaningful features without any label information. A typical CL framework is divided into two phases, where it first tries to learn the features from unlabelled data, and then uses those features to train a linear classifier with the labeled data. While a fair amount of existing theoretical works have analyzed how the unsupervised loss in the first phase can support the supervised loss in the second phase, none has examined the connection between the unsupervised loss and the robust supervised loss, which can shed light on how to construct an effective unsupervised loss for the first phase of CL. To fill this gap, our work develops rigorous theories to dissect and identify which components in the unsupervised loss can help improve the robust supervised loss and conduct proper experiments to verify our findings.
Code (0)
등록된 구현이 없습니다.
Tasks
Contrastive LearningSimilar Papers 제목 키워드 기반
Chaos is a Ladder: A New Understanding of Contrastive Learning
Recently, contrastive learning has risen to be a promising approach for large-scale self-supervised learning. However, theoretical understanding of how it works is still unclear. In this paper, we propose a new guarantee…
Contrastive LearningSelf-Supervised LearningConvergence of energy-based learning in linear resistive networks
Energy-based learning algorithms are alternatives to backpropagation and are well-suited to distributed implementations in analog electronic devices. However, a rigorous theory of convergence is lacking. We make a first …
Contrastive LearningChaos is a Ladder: A New Theoretical Understanding of Contrastive Learning via Augmentation Overlap
Recently, contrastive learning has risen to be a promising approach for large-scale self-supervised learning. However, theoretical understanding of how it works is still unclear. In this paper, we propose a new guarantee…
ARCContrastive LearningModel SelectionSelf-Supervised LearningTowards Understanding the Mechanism of Contrastive Learning via Similarity Structure: A Theoretical Analysis
Contrastive learning is an efficient approach to self-supervised representation learning. Although recent studies have made progress in the theoretical understanding of contrastive learning, the investigation of how to c…
Contrastive LearningRepresentation LearningA Statistical Theory of Contrastive Pre-training and Multimodal Generative AI
Multi-modal generative AI systems, such as those combining vision and language, rely on contrastive pre-training to learn representations across different modalities. While their practical benefits are widely acknowledge…
zero-shot-classificationZero-Shot Learning