Dual-Alignment Pre-training for Cross-lingual Sentence Embedding
Recent studies have shown that dual encoder models trained with the sentence-level translation ranking task are effective methods for cross-lingual sentence embedding. However, our research indicates that token-level alignment is also crucial in multilingual scenarios, which has not been fully explored previously. Based on our findings, we propose a dual-alignment pre-training (DAP) framework for cross-lingual sentence embedding that incorporates both sentence-level and token-level alignment. To achieve this, we introduce a novel representation translation learning (RTL) task, where the model learns to use one-side contextualized token representation to reconstruct its translation counterpart. This reconstruction objective encourages the model to embed translation information into the token representation. Compared to other token-level alignment methods such as translation language modeling, RTL is more suitable for dual encoder architectures and is computationally efficient. Extensive experiments on three sentence-level cross-lingual benchmarks demonstrate that our approach can significantly improve sentence embedding. Our code is available at https://github.com/ChillingDream/DAP.
Code (1)
Tasks
Language ModelingLanguage ModellingSentenceSentence EmbeddingSentence-EmbeddingTranslationSimilar Papers 제목 키워드 기반
Modular Sentence Encoders: Separating Language Specialization from Cross-Lingual Alignment
Multilingual sentence encoders are commonly obtained by training multilingual language models to map sentences from different languages into a shared semantic space. As such, they are subject to curse of multilinguality,…
Contrastive LearningMultiple-choiceSentenceSentence Embeddings+1Bilingual alignment transfers to multilingual alignment for unsupervised parallel text mining
This work presents methods for learning cross-lingual sentence representations using paired or unpaired bilingual texts. We hypothesize that the cross-lingual alignment strategy is transferable, and therefore a model tra…
Language ModelingLanguage ModellingRetrievalSentence+1Lightweight Cross-Lingual Sentence Representation Learning
Large-scale models for learning fixed-dimensional cross-lingual sentence representations like LASER (Artetxe and Schwenk, 2019b) lead to significant improvement in performance on downstream tasks. However, further increa…
Contrastive LearningDocument ClassificationLanguage ModelingLanguage Modelling+4Alibaba Submission to the WMT20 Parallel Corpus Filtering Task
This paper describes the Alibaba Machine Translation Group submissions to the WMT 2020 Shared Task on Parallel Corpus Filtering and Alignment. In the filtering task, three main methods are applied to evaluate the quality…
DiversityLanguage IdentificationMachine TranslationSentence+3Improving In-context Learning of Multilingual Generative Language Models with Cross-lingual Alignment
Multilingual generative models obtain remarkable cross-lingual in-context learning capabilities through pre-training on large-scale corpora. However, they still exhibit a performance bias toward high-resource languages a…
Contrastive LearningIn-Context LearningSentenceTransfer Learning