SimCSE++: Improving Contrastive Learning for Sentence Embeddings from Two Perspectives
This paper improves contrastive learning for sentence embeddings from two perspectives: handling dropout noise and addressing feature corruption. Specifically, for the first perspective, we identify that the dropout noise from negative pairs affects the model's performance. Therefore, we propose a simple yet effective method to deal with such type of noise. Secondly, we pinpoint the rank bottleneck of current solutions to feature corruption and propose a dimension-wise contrastive learning objective to address this issue. Both proposed methods are generic and can be applied to any contrastive learning based models for sentence embeddings. Experimental results on standard benchmarks demonstrate that combining both proposed methods leads to a gain of 1.8 points compared to the strong baseline SimCSE configured with BERT base. Furthermore, applying the proposed method to DiffCSE, another strong contrastive learning based baseline, results in a gain of 1.4 points.
Code (0)
등록된 구현이 없습니다.
Tasks
Contrastive LearningSentenceSentence EmbeddingsMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
English Contrastive Learning Can Learn Universal Cross-lingual Sentence Embeddings
Universal cross-lingual sentence embeddings map semantically similar cross-lingual sentences into a shared embedding space. Aligning cross-lingual sentence embeddings usually requires supervised cross-lingual parallel se…
Contrastive LearningRetrievalSentenceSentence Embedding+3Improving Contrastive Learning of Sentence Embeddings with Focal-InfoNCE
The recent success of SimCSE has greatly advanced state-of-the-art sentence representations. However, the original formulation of SimCSE does not fully exploit the potential of hard negative samples in contrastive learni…
Contrastive LearningSentenceSentence EmbeddingsSTSNon-contrastive sentence representations via self-supervision
Sample contrastive methods, typically referred to simply as contrastive are the foundation of most unsupervised methods to learn text and sentence embeddings. On the other hand, a different class of self-supervised loss …
SentenceSentence EmbeddingsESimCSE: Enhanced Sample Building Method for Contrastive Learning of Unsupervised Sentence Embedding
Contrastive learning has been attracting much attention for learning unsupervised sentence embeddings. The current state-of-the-art unsupervised method is the unsupervised SimCSE (unsup-SimCSE). Unsup-SimCSE takes dropou…
Contrastive LearningData AugmentationSentenceSentence Embedding+4S-SimCSE: Sampled Sub-networks for Contrastive Learning of Sentence Embedding
Contrastive learning has been studied for improving the performance of learning sentence embeddings. The current state-of-the-art method is the SimCSE, which takes dropout as the data augmentation method and feeds a pre-…
Contrastive LearningData AugmentationSentenceSentence Embedding+3