Bitext Mining for Low-Resource Languages via Contrastive Learning
Mining high-quality bitexts for low-resource languages is challenging. This paper shows that sentence representation of language models fine-tuned with multiple negatives ranking loss, a contrastive objective, helps retrieve clean bitexts. Experiments show that parallel data mined from our approach substantially outperform the previous state-of-the-art method on low resource languages Khmer and Pashto.
Code (1)
Tasks
Contrastive LearningSentenceSimilar Papers 제목 키워드 기반
xSIM++: An Improved Proxy to Bitext Mining Performance for Low-Resource Languages
We introduce a new proxy score for evaluating bitext mining based on similarity in a multilingual embedding space: xSIM++. In comparison to xSIM, this improved proxy leverages rule-based approaches to extend English sent…
NMTBitext Mining Using Distilled Sentence Representations for Low-Resource Languages
Scaling multilingual representation learning beyond the hundred most frequent languages is challenging, in particular to cover the long tail of low-resource languages. A promising approach has been to train one-for-all m…
Cross-Lingual TransferNMTRepresentation LearningSentenceCross-Lingual Representation Alignment Through Contrastive Image-Caption Tuning
Multilingual alignment of sentence representations has mostly required bitexts to bridge the gap between languages. We investigate whether visual information can bridge this gap instead. Image caption datasets are very e…
Natural Language UnderstandingRetrievalSentenceEnhancing Multilingual Embeddings via Multi-Way Parallel Text Alignment
Multilingual pretraining typically lacks explicit alignment signals, leading to suboptimal cross-lingual alignment in the representation space. In this work, we show that training standard pretrained models for cross-lin…
Contrastive LearningSemantic SimilarityWikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia
We present an approach based on multilingual sentence embeddings to automatically extract parallel sentences from the content of Wikipedia articles in 85 languages, including several dialects or low-resource languages. W…
ArticlesSentenceSentence Embeddings