InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddings
Contrastive learning has been extensively studied in sentence embedding learning, which assumes that the embeddings of different views of the same sentence are closer. The constraint brought by this assumption is weak, and a good sentence representation should also be able to reconstruct the original sentence fragments. Therefore, this paper proposes an information-aggregated contrastive learning framework for learning unsupervised sentence embeddings, termed InfoCSE. InfoCSE forces the representation of [CLS] positions to aggregate denser sentence information by introducing an additional Masked language model task and a well-designed network. We evaluate the proposed InfoCSE on several benchmark datasets w.r.t the semantic text similarity (STS) task. Experimental results show that InfoCSE outperforms SimCSE by an average Spearman correlation of 2.60% on BERT-base, and 1.77% on BERT-large, achieving state-of-the-art results among unsupervised sentence representation learning methods. Our code are available at https://github.com/caskcsg/sentemb/tree/main/InfoCSE.
Code (2)
Tasks
Contrastive LearningLanguage ModelingLanguage ModellingRepresentation LearningSentenceSentence EmbeddingSentence-EmbeddingSentence EmbeddingsSTStext similarityMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
A Mutually Reinforced Framework for Pretrained Sentence Embeddings
The lack of labeled data is a major obstacle to learning high-quality sentence embeddings. Recently, self-supervised contrastive learning (SCL) is regarded as a promising way to address this problem. However, the existin…
Contrastive LearningRepresentation LearningSentenceSentence EmbeddingsInstance Smoothed Contrastive Learning for Unsupervised Sentence Embedding
Contrastive learning-based methods, such as unsup-SimCSE, have achieved state-of-the-art (SOTA) performances in learning unsupervised sentence embeddings. However, in previous studies, each embedding used for contrastive…
Contrastive LearningSemantic SimilaritySemantic Textual SimilaritySentence+5miCSE: Mutual Information Contrastive Learning for Low-shot Sentence Embeddings
This paper presents miCSE, a mutual information-based contrastive learning framework that significantly advances the state-of-the-art in few-shot sentence embedding. The proposed approach imposes alignment between the at…
Contrastive LearningFew-Shot LearningSelf-Supervised LearningSentence+3Contrastive Learning of Sentence Representations
Learning sentence representations which capture rich semantic meanings has been crucial for many NLP tasks. Pre-trained language models such as BERT have achieved great success in NLP, but sentence embeddings extracted d…
Contrastive LearningSemantic SimilaritySemantic Textual SimilaritySentence+1Span-Aggregatable, Contextualized Word Embeddings for Effective Phrase Mining
Dense vector representations for sentences made significant progress in recent years as can be seen on sentence similarity tasks. Real-world phrase retrieval applications, on the other hand, still encounter challenges fo…
RetrievalSentenceSentence EmbeddingsSentence Similarity+3