Training Effective Neural Sentence Encoders from Automatically Mined Paraphrases
Sentence embeddings are commonly used in text clustering and semantic retrieval tasks. State-of-the-art sentence representation methods are based on artificial neural networks fine-tuned on large collections of manually labeled sentence pairs. Sufficient amount of annotated data is available for high-resource languages such as English or Chinese. In less popular languages, multilingual models have to be used, which offer lower performance. In this publication, we address this problem by proposing a method for training effective language-specific sentence encoders without manually labeled data. Our approach is to automatically construct a dataset of paraphrase pairs from sentence-aligned bilingual text corpora. We then use the collected data to fine-tune a Transformer language model with an additional recurrent pooling layer. Our sentence encoder can be trained in less than a day on a single graphics card, achieving high performance on a diverse set of sentence-level tasks. We evaluate our method on eight linguistic tasks in Polish, comparing it with the best available multilingual sentence encoders.
Code (1)
Tasks
Language ModelingLanguage ModellingRetrievalSemantic RetrievalSentenceSentence EmbeddingsText ClusteringMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Fake Sentence Detection as a Training Task for Sentence Encoding
Sentence encoders are typically trained on language modeling tasks with large unlabeled datasets. While these encoders achieve state-of-the-art results on many sentence-level tasks, they are difficult to train with long …
Binary ClassificationLanguage ModelingLanguage ModellingSentenceFake Sentence Detection as a Training Task for Sentence Encoding
Sentence encoders are typically trained on language modeling tasks which enable them to use large unlabeled datasets. While these models achieve state-of-the-art results on many sentence-level tasks, they are difficult …
Binary ClassificationLanguage ModelingLanguage ModellingSentenceModular Sentence Encoders: Separating Language Specialization from Cross-Lingual Alignment
Multilingual sentence encoders are commonly obtained by training multilingual language models to map sentences from different languages into a shared semantic space. As such, they are subject to curse of multilinguality,…
Contrastive LearningMultiple-choiceSentenceSentence Embeddings+1Ranking-Enhanced Unsupervised Sentence Representation Learning
Unsupervised sentence representation learning has progressed through contrastive learning and data augmentation methods such as dropout masking. Despite this progress, sentence encoders are still limited to using only an…
Contrastive LearningData AugmentationRepresentation LearningSemantic Similarity+4DisSent: Sentence Representation Learning from Explicit Discourse Relations
Learning effective representations of sentences is one of the core missions of natural language understanding. Existing models either train on a vast amount of text, or require costly, manually curated sentence relation …
Dependency ParsingNatural Language UnderstandingRelationRelation Prediction+3