Less can be more in contrastive learning
Unsupervised representation learning provides an attractive alternative to its supervised counterpart because of the abundance of unlabelled data. Contrastive learning has recently emerged as one of the most successful approaches to unsupervised representation learning. Given a datapoint, contrastive learning involves discriminating between a matching, or positive, datapoint and a number of non-matching, or negative, ones. Usually the other datapoints in the batch serve as the negatives for the given datapoint. It has been shown empirically that large batch sizes are needed to achieve good performance, which led the the belief that a large number of negatives is preferable. In order to understand this phenomenon better, in this work investigate the role of negatives in contrastive learning by decoupling the number of negatives from the batch size. Surprisingly, we discover that for a fixed batch size performance actually degrades as the number of negatives is increased. We also show that using fewer negatives can lead to a better signal-to-noise ratio for the model gradients, which could explain the improved performance.
Code (0)
등록된 구현이 없습니다.
Tasks
Contrastive LearningRepresentation LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Quantifying and Mitigating Privacy Risks of Contrastive Learning
Data is the key factor to drive the development of machine learning (ML) during the past decade. However, high-quality data, in particular labeled data, is often hard and expensive to collect. To leverage large-scale unl…
AttributeBIG-bench Machine LearningContrastive LearningData Augmentation+2A Primer on Contrastive Pretraining in Language Processing: Methods, Lessons Learned and Perspectives
Modern natural language processing (NLP) methods employ self-supervised pretraining objectives such as masked language modeling to boost the performance of various application tasks. These pretraining methods are frequen…
Contrastive LearningLanguage ModelingLanguage ModellingMasked Language Modeling+3Not All Negatives are Equal: Label-Aware Contrastive Loss for Fine-grained Text Classification
Fine-grained classification involves dealing with datasets with larger number of classes with subtle differences between them. Guiding the model to focus on differentiating dimensions between these commonly confusable cl…
AllClassificationEmotion ClassificationSentiment Analysis+2HILL: Hierarchy-aware Information Lossless Contrastive Learning for Hierarchical Text Classification
Existing self-supervised methods in natural language processing (NLP), especially hierarchical text classification (HTC), mainly focus on self-supervised contrastive learning, extremely relying on human-designed augmenta…
Contrastive LearningDocument EmbeddingHierarchical Multi-label ClassificationRepresentation Learning+2Improving BERT Model Using Contrastive Learning for Biomedical Relation Extraction
Contrastive learning has been used to learn a high-quality representation of the image in computer vision. However, contrastive learning is not widely utilized in natural language processing due to the lack of a general …
Contrastive LearningData AugmentationRelationRelation Extraction