On Isotropy, Contextualization and Learning Dynamics of Contrastive-based Sentence Representation Learning
Incorporating contrastive learning objectives in sentence representation learning (SRL) has yielded significant improvements on many sentence-level NLP tasks. However, it is not well understood why contrastive learning works for learning sentence-level semantics. In this paper, we aim to help guide future designs of sentence representation learning methods by taking a closer look at contrastive SRL through the lens of isotropy, contextualization and learning dynamics. We interpret its successes through the geometry of the representation shifts and show that contrastive learning brings isotropy, and drives high intra-sentence similarity: when in the same sentence, tokens converge to similar positions in the semantic space. We also find that what we formalize as "spurious contextualization" is mitigated for semantically meaningful tokens, while augmented for functional ones. We find that the embedding space is directed towards the origin during training, with more areas now better defined. We ablate these findings by observing the learning dynamics with different training temperatures, batch sizes and pooling methods.
Code (1)
Tasks
Contrastive LearningRepresentation LearningSentenceSentence SimilarityMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Contrastive Visual Semantic Pretraining Magnifies the Semantics of Natural Language Representations
We examine the effects of contrastive visual semantic pretraining by comparing the geometry and semantic properties of contextualized English language representations formed by GPT-2 and CLIP, a zero-shot multimodal imag…
Image CaptioningSemantic Textual SimilaritySentenceSentence Embeddings+1Debiased Contrastive Learning of Unsupervised Sentence Representations
Recently, contrastive learning has shown effectiveness in fine-tuning pre-trained language models (PLM) to derive sentence representations, which pulls augmented positive examples together to improve the alignment while …
Contrastive LearningSemantic Textual SimilaritySentenceWhitening Sentence Representations for Better Semantics and Faster Retrieval
Pre-training models such as BERT have achieved great success in many natural language processing tasks. However, how to obtain better sentence representation through these pre-training models is still worthy to exploit. …
RetrievalSentenceDebiased Contrastive Learning of Unsupervised Sentence Representations
Recently, contrastive learning has been shown to be effective in improving pre-trained language models (PLM) to derive high-quality sentence representations. It aims to pull close positive examples to enhance the alignme…
Contrastive LearningSemantic Textual SimilaritySentenceHiCLRE: A Hierarchical Contrastive Learning Framework for Distantly Supervised Relation Extraction
Distant supervision assumes that any sentence containing the same entity pairs reflects identical relationships. Previous works of distantly supervised relation extraction (DSRE) task generally focus on sentence-level or…
Contrastive LearningData AugmentationRelationRelation Extraction+1