CLEAR: Contrastive Learning for Sentence Representation
Pre-trained language models have proven their unique powers in capturing implicit language features. However, most pre-training approaches focus on the word-level training objective, while sentence-level objectives are rarely studied. In this paper, we propose Contrastive LEArning for sentence Representation (CLEAR), which employs multiple sentence-level augmentation strategies in order to learn a noise-invariant sentence representation. These augmentations include word and span deletion, reordering, and substitution. Furthermore, we investigate the key reasons that make contrastive learning effective through numerous experiments. We observe that different sentence augmentations during pre-training lead to different performance improvements on various downstream tasks. Our approach is shown to outperform multiple existing methods on both SentEval and GLUE benchmarks.
Code (0)
등록된 구현이 없습니다.
Tasks
Contrastive LearningLinguistic AcceptabilityNatural Language InferenceQuestion AnsweringSemantic Textual SimilaritySentenceSentiment AnalysisMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Self-Guided Contrastive Learning for BERT Sentence Representations
Although BERT and its variants have reshaped the NLP landscape, it still remains unclear how best to derive sentence embeddings from such pre-trained Transformers. In this work, we propose a contrastive learning method t…
Contrastive LearningData AugmentationRepresentation LearningSentence+1Hyper-CL: Conditioning Sentence Representations with Hypernetworks
While the introduction of contrastive learning frameworks in sentence representation learning has significantly contributed to advancements in the field, it still remains unclear whether state-of-the-art sentence embeddi…
Computational EfficiencyContrastive LearningKnowledge Graph CompletionRepresentation Learning+3Towards Better Understanding of Contrastive Sentence Representation Learning: A Unified Paradigm for Gradient
Sentence Representation Learning (SRL) is a crucial task in Natural Language Processing (NLP), where contrastive Self-Supervised Learning (SSL) is currently a mainstream approach. However, the reasons behind its remarkab…
Representation LearningSelf-Supervised LearningSemantic Textual SimilaritySentence+1Don't Judge a Language Model by Its Last Layer: Contrastive Learning with Layer-Wise Attention Pooling
Recent pre-trained language models (PLMs) achieved great success on many natural language processing tasks through learning linguistic features and contextualized sentence representation. Since attributes captured in sta…
Contrastive LearningLanguage ModelingLanguage ModellingSemantic Textual Similarity+2Don’t Judge a Language Model by Its Last Layer: Contrastive Learning with Layer-Wise Attention Pooling
Recent pre-trained language models (PLMs) achieved great success on many natural language processing tasks through learning linguistic features and contextualized sentence representation. Since attributes captured in sta…
Contrastive LearningLanguage ModelingLanguage ModellingSemantic Textual Similarity+2