Cross-lingual Pre-training Based Transfer for Zero-shot Neural Machine Translation
Transfer learning between different language pairs has shown its effectiveness for Neural Machine Translation (NMT) in low-resource scenario. However, existing transfer methods involving a common target language are far from success in the extreme scenario of zero-shot translation, due to the language space mismatch problem between transferor (the parent model) and transferee (the child model) on the source side. To address this challenge, we propose an effective transfer learning approach based on cross-lingual pre-training. Our key idea is to make all source languages share the same feature space and thus enable a smooth transition for zero-shot translation. To this end, we introduce one monolingual pre-training method and two bilingual pre-training methods to obtain a universal encoder for different languages. Once the universal encoder is constructed, the parent model built on such encoder is trained with large-scale annotated data and then directly applied in zero-shot translation scenario. Experiments on two public datasets show that our approach significantly outperforms strong pivot-based baseline and various multilingual NMT approaches.
Code (0)
등록된 구현이 없습니다.
Tasks
Machine TranslationNMTTransfer LearningTranslationSimilar Papers 제목 키워드 기반
Improving Zero-Shot Cross-Lingual Transfer Learning via Robust Training
Pre-trained multilingual language encoders, such as multilingual BERT and XLM-R, show great potential for zero-shot cross-lingual transfer. However, these multilingual encoders do not precisely align words and phrases ac…
Cross-Lingual TransferSentencetext-classificationText Classification+3Self-Augmentation Improves Zero-Shot Cross-Lingual Transfer
Zero-shot cross-lingual transfer is a central task in multilingual NLP, allowing models trained in languages with more sufficient training resources to generalize to other low-resource languages. Earlier efforts on this …
Cross-Lingual TransferMultilingual NLPZero-Shot Cross-Lingual TransferZero-shot Reading Comprehension by Cross-lingual Transfer Learning with Multi-lingual Language Representation Model
Because it is not feasible to collect training data for every language, there is a growing interest in cross-lingual transfer learning. In this paper, we systematically explore zero-shot cross-lingual transfer learning o…
Cross-Lingual TransferReading ComprehensionTransfer LearningZero-Shot Cross-Lingual Transfer+1Bilingual Alignment Pre-Training for Zero-Shot Cross-Lingual Transfer
Multilingual pre-trained models have achieved remarkable performance on cross-lingual transfer learning. Some multilingual models such as mBERT, have been pre-trained on unlabeled corpora, therefore the embeddings of dif…
Cross-Lingual TransferLanguage ModellingMachine Reading ComprehensionReading Comprehension+2Cross-Lingual Text Classification with Multilingual Distillation and Zero-Shot-Aware Training
Multilingual pre-trained language models (MPLMs) not only can handle tasks in different languages but also exhibit surprising zero-shot cross-lingual transferability. However, MPLMs usually are not able to achieve compar…
Language ModelingLanguage Modellingtext-classificationText Classification