The Impact of Cross-Lingual Adjustment of Contextual Word Representations on Zero-Shot Transfer
Large pre-trained multilingual models such as mBERT and XLM-R enabled effective cross-lingual zero-shot transfer in many NLP tasks. A cross-lingual adjustment of these models using a small parallel corpus can further improve results. This is a more data efficient method compared to training a machine-translation system or a multi-lingual model from scratch using only parallel data. In this study, we experiment with zero-shot transfer of English models to four typologically different languages (Spanish, Russian, Vietnamese, and Hindi) and three NLP tasks (QA, NLI, and NER). We carry out a cross-lingual adjustment of an off-the-shelf mBERT model. We show that this adjustment makes embeddings of semantically similar words from different languages closer to each other, while keeping unrelated words apart. In contrast, fine-tuning of mBERT on English data (for a specific task such as NER) draws embeddings of both related and unrelated words closer to each other. The cross-lingual adjustment of mBERT improves NLI in four languages and NER in two languages. However, in the case of QA performance never improves and sometimes degrades. In that, the increase in the amount of parallel data is most beneficial for NLI, whereas QA performance peaks at roughly 5K parallel sentences and further decreases as the number of parallel sentences increases.
Code (0)
등록된 구현이 없습니다.
Tasks
Machine TranslationNERXLM-RMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
The Impact of Cross-Lingual Adjustment of Contextual Word Representations on Zero-Shot Transfer
Large multilingual language models such as mBERT or XLM-R enable zero-shot cross-lingual transfer in various IR and NLP tasks. Cao et al. (2020) proposed a data- and compute-efficient method for cross-lingual adjustment …
Continual LearningCross-Lingual TransferMachine Reading ComprehensionNatural Language Inference+3Cross-lingual Semantic Role Labeling with Model Transfer
Prior studies show that cross-lingual semantic role labeling (SRL) can be achieved by model transfer under the help of universal features. In this paper, we fill the gap of cross-lingual SRL by proposing an end-to-end SR…
Semantic Role LabelingCross-Lingual Contextual Word Embeddings Mapping With Multi-Sense Words In Mind
Recent work in cross-lingual contextual word embedding learning cannot handle multi-sense words well. In this work, we explore the characteristics of contextual word embeddings and show the link between contextual word e…
Bilingual Lexicon InductionWord EmbeddingsMeasure and Evaluation of Semantic Divergence across Two Languages
Languages are dynamic systems: word usage may change over time, reflecting various societal factors. However, all languages do not evolve identically: the impact of an event, the influence of a trend or thinking, can dif…
TranslationVocal Bursts Valence PredictionWord EmbeddingsImproving Zero-Shot Cross-Lingual Transfer via Progressive Code-Switching
Code-switching is a data augmentation scheme mixing words from multiple languages into source lingual text. It has achieved considerable generalization performance of cross-lingual transfer tasks by aligning cross-lingua…
Cross-Lingual TransferData AugmentationModel OptimizationSentence+1