Do More Negative Samples Necessarily Hurt in Contrastive Learning?
Recent investigations in noise contrastive estimation suggest, both empirically as well as theoretically, that while having more "negative samples" in the contrastive loss improves downstream classification performance initially, beyond a threshold, it hurts downstream performance due to a "collision-coverage" trade-off. But is such a phenomenon inherent in contrastive learning? We show in a simple theoretical setting, where positive pairs are generated by sampling from the underlying latent class (introduced by Saunshi et al. (ICML 2019)), that the downstream performance of the representation optimizing the (population) contrastive loss in fact does not degrade with the number of negative samples. Along the way, we give a structural characterization of the optimal representation in our framework, for noise contrastive estimation. We also provide empirical support for our theoretical results on CIFAR-10 and CIFAR-100 datasets.
Code (0)
등록된 구현이 없습니다.
Tasks
Contrastive LearningSimilar Papers 제목 키워드 기반
Contrast Is All You Need
In this study, we analyze data-scarce classification scenarios, where available labeled legal data is small and imbalanced, potentially hurting the quality of the results. We focused on two finetuning objectives; SetFit …
AllClassificationContrastive LearningSentenceSynthetic Hard Negative Samples for Contrastive Learning
Contrastive learning has emerged as an essential approach for self-supervised learning in visual representation learning. The central objective of contrastive learning is to maximize the similarities between two augmente…
Contrastive LearningRepresentation LearningSelf-Supervised LearningConCur: Self-supervised graph representation based on contrastive learning with curriculum negative sampling
Contrastive learning has made breakthrough advancements in graph representation learning, which encourages the representation of positive samples to be close and those of negative samples to be far away. However, existin…
Contrastive LearningGraph Representation LearningNode ClassificationRepresentation Learning+1Debiased Contrastive Learning of Unsupervised Sentence Representations
Recently, contrastive learning has shown effectiveness in fine-tuning pre-trained language models (PLM) to derive sentence representations, which pulls augmented positive examples together to improve the alignment while …
Contrastive LearningSemantic Textual SimilaritySentenceRethinking Samples Selection for Contrastive Learning: Mining of Potential Samples
Contrastive learning predicts whether two images belong to the same category by training a model to make their feature representations as close or as far away as possible. In this paper, we rethink how to mine samples in…
Contrastive LearningData Augmentation