ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
Learning high-quality sentence representations benefits a wide range of natural language processing tasks. Though BERT-based pre-trained language models achieve high performance on many downstream tasks, the native derived sentence representations are proved to be collapsed and thus produce a poor performance on the semantic textual similarity (STS) tasks. In this paper, we present ConSERT, a Contrastive Framework for Self-Supervised Sentence Representation Transfer, that adopts contrastive learning to fine-tune BERT in an unsupervised and effective way. By making use of unlabeled texts, ConSERT solves the collapse issue of BERT-derived sentence representations and make them more applicable for downstream tasks. Experiments on STS datasets demonstrate that ConSERT achieves an 8\% relative improvement over the previous state-of-the-art, even comparable to the supervised SBERT-NLI. And when further incorporating NLI supervision, we achieve new state-of-the-art performance on STS tasks. Moreover, ConSERT obtains comparable results with only 1000 samples available, showing its robustness in data scarcity scenarios.
Code (1)
Tasks
Contrastive LearningSemantic Textual SimilaritySentenceSTSMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
DebCSE: Rethinking Unsupervised Contrastive Sentence Embedding Learning in the Debiasing Perspective
Several prior studies have suggested that word frequency biases can cause the Bert model to learn indistinguishable sentence embeddings. Contrastive learning schemes such as SimCSE and ConSERT have already been adopted s…
Contrastive LearningSemantic SimilaritySemantic Textual SimilaritySentence+4PromptBERT: Improving BERT Sentence Embeddings with Prompts
The poor performance of the original BERT for sentence semantic similarity has been widely discussed in previous works. We find that unsatisfactory performance is mainly due to the static token embeddings biases and th…
DenoisingSemantic SimilaritySemantic Textual SimilaritySentence+2reCSE: Portable Reshaping Features for Sentence Embedding in Self-supervised Contrastive Learning
We propose reCSE, a self supervised contrastive learning sentence representation framework based on feature reshaping. This framework is different from the current advanced models that use discrete data augmentation meth…
Contrastive LearningData AugmentationGPUSemantic Similarity+4Non-contrastive sentence representations via self-supervision
Sample contrastive methods, typically referred to simply as contrastive are the foundation of most unsupervised methods to learn text and sentence embeddings. On the other hand, a different class of self-supervised loss …
SentenceSentence EmbeddingsmiCSE: Mutual Information Contrastive Learning for Low-shot Sentence Embeddings
This paper presents miCSE, a mutual information-based contrastive learning framework that significantly advances the state-of-the-art in few-shot sentence embedding. The proposed approach imposes alignment between the at…
Contrastive LearningFew-Shot LearningSelf-Supervised LearningSentence+3