TransAug: Translate as Augmentation for Sentence Embeddings
While contrastive learning greatly advances the representation of sentence embeddings, it is still limited by the size of the existing sentence datasets. In this paper, we present TransAug (Translate as Augmentation), which provide the first exploration of utilizing translated sentence pairs as data augmentation for text, and introduce a two-stage paradigm to advances the state-of-the-art sentence embeddings. Instead of adopting an encoder trained in other languages setting, we first distill a Chinese encoder from a SimCSE encoder (pretrained in English), so that their embeddings are close in semantic space, which can be regraded as implicit data augmentation. Then, we only update the English encoder via cross-lingual contrastive learning and frozen the distilled Chinese encoder. Our approach achieves a new state-of-art on standard semantic textual similarity (STS), outperforming both SimCSE and Sentence-T5, and the best performance in corresponding tracks on transfer tasks evaluated by SentEval.
Code (0)
등록된 구현이 없습니다.
Tasks
Contrastive LearningData AugmentationSemantic Textual SimilaritySentenceSentence EmbeddingsSTSMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Back Translation Survey for Improving Text Augmentation
Natural Language Processing (NLP) relies heavily on training data. Transformers, as they have gotten bigger, have required massive amounts of training data. To satisfy this requirement, text augmentation should be looked…
SentenceSurveyText AugmentationTranslationMaking Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
We present an easy and efficient method to extend existing sentence embedding models to new languages. This allows to create multilingual versions from previously monolingual models. The training is based on the idea tha…
Knowledge DistillationSentenceSentence EmbeddingSentence-Embedding+1DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
We propose DiffCSE, an unsupervised contrastive learning framework for learning sentence embeddings. DiffCSE learns sentence embeddings that are sensitive to the difference between the original sentence and an edited sen…
Contrastive LearningLanguage ModelingLanguage ModellingRepresentation Learning+3PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
Learning sentence embeddings in an unsupervised manner is fundamental in natural language processing. Recent common practice is to couple pre-trained language models with unsupervised contrastive learning, whose success …
Contrastive LearningOpen-Ended Question AnsweringSentenceSentence Embeddings+1GASE: Generatively Augmented Sentence Encoding
We propose an approach to enhance sentence embeddings by applying generative text models for data augmentation at inference time. Unlike conventional data augmentation that utilises synthetic training data, our approach …
Data AugmentationSemantic Textual SimilaritySentenceSentence Embeddings+1