Bilingual Alignment Pre-Training for Zero-Shot Cross-Lingual Transfer
Multilingual pre-trained models have achieved remarkable performance on cross-lingual transfer learning. Some multilingual models such as mBERT, have been pre-trained on unlabeled corpora, therefore the embeddings of different languages in the models may not be aligned very well. In this paper, we aim to improve the zero-shot cross-lingual transfer performance by proposing a pre-training task named Word-Exchange Aligning Model (WEAM), which uses the statistical alignment information as the prior knowledge to guide cross-lingual word prediction. We evaluate our model on multilingual machine reading comprehension task MLQA and natural language interface task XNLI. The results show that WEAM can significantly improve the zero-shot performance.
Code (1)
Tasks
Cross-Lingual TransferLanguage ModellingMachine Reading ComprehensionReading ComprehensionTransfer LearningZero-Shot Cross-Lingual TransferMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Zero-shot Cross-lingual Transfer without Parallel Corpus
Recently, although pre-trained language models have achieved great success on multilingual NLP (Natural Language Processing) tasks, the lack of training data on many tasks in low-resource languages still limits their per…
Cross-Lingual TransferMultilingual NLPTranslationZero-Shot Cross-Lingual TransferSelf-Augmentation Improves Zero-Shot Cross-Lingual Transfer
Zero-shot cross-lingual transfer is a central task in multilingual NLP, allowing models trained in languages with more sufficient training resources to generalize to other low-resource languages. Earlier efforts on this …
Cross-Lingual TransferMultilingual NLPZero-Shot Cross-Lingual TransferA Simple and Effective Method to Improve Zero-Shot Cross-Lingual Transfer Learning
Existing zero-shot cross-lingual transfer methods rely on parallel corpora or bilingual dictionaries, which are expensive and impractical for low-resource languages. To disengage from these dependencies, researchers have…
Cross-Lingual Transfertext-classificationText ClassificationTransfer Learning+2Bilingual Text-to-Motion Generation: A New Benchmark and Baselines
Text-to-motion generation holds significant potential for cross-linguistic applications, yet it is hindered by the lack of bilingual datasets and the poor cross-lingual semantic understanding of existing language models.…
Motion SynthesisZero-shot Cross-lingual Transfer is Under-specified Optimization
Pretrained multilingual encoders enable zero-shot cross-lingual transfer, but often produce unreliable models that exhibit high performance variance on the target language. We postulate that this high variance results fr…
Cross-Lingual TransferZero-Shot Cross-Lingual Transfer