An Efficient Self-Supervised Cross-View Training For Sentence Embedding
Self-supervised sentence representation learning is the task of constructing an embedding space for sentences without relying on human annotation efforts. One straightforward approach is to finetune a pretrained language model (PLM) with a representation learning method such as contrastive learning. While this approach achieves impressive performance on larger PLMs, the performance rapidly degrades as the number of parameters decreases. In this paper, we propose a framework called Self-supervised Cross-View Training (SCT) to narrow the performance gap between large and small PLMs. To evaluate the effectiveness of SCT, we compare it to 5 baseline and state-of-the-art competitors on seven Semantic Textual Similarity (STS) benchmarks using 5 PLMs with the number of parameters ranging from 4M to 340M. The experimental results show that STC outperforms the competitors for PLMs with less than 100M parameters in 18 of 21 cases.
Code (1)
Tasks
Contrastive LearningLanguage ModelingLanguage ModellingRepresentation LearningSemantic Textual SimilaritySentenceSentence EmbeddingSentence-EmbeddingSTSMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Improving Sentence Representations with Consensus Maximisation
Consensus maximisation learning can provide self-supervision when different views are available of the same data. The distributional hypothesis provides another form of useful self-supervision from adjacent sentences whi…
Self-Supervised LearningSentencemiCSE: Mutual Information Contrastive Learning for Low-shot Sentence Embeddings
This paper presents miCSE, a mutual information-based contrastive learning framework that significantly advances the state-of-the-art in few-shot sentence embedding. The proposed approach imposes alignment between the at…
Contrastive LearningFew-Shot LearningSelf-Supervised LearningSentence+3Sentence embedding with contrastive multi-views learning
In this work, we propose a self-supervised method to learn sentence representations with an injection of linguistic knowledge. Multiple linguistic frameworks propose diverse sentence structures from which semantic meanin…
DiversitySentenceSentence EmbeddingSentence-EmbeddingRevisiting Self-Training for Few-Shot Learning of Language Model
As unlabeled data carry rich task-relevant information, they are proven useful for few-shot learning of language model. The question is how to effectively make use of such data. In this work, we revisit the self-training…
BenchmarkingFew-Shot LearningLanguage ModelingLanguage Modelling+5Semi-supervised Thai Sentence Segmentation Using Local and Distant Word Representations
A sentence is typically treated as the minimal syntactic unit used for extracting valuable information from a longer piece of text. However, in written Thai, there are no explicit sentence markers. We proposed a deep lea…
SentenceSentence segmentation