Analyzing and Reducing the Performance Gap in Cross-Lingual Transfer with Fine-tuning Slow and Fast
Existing research has shown that a multilingual pre-trained language model fine-tuned with one (source) language also performs well on downstream tasks for non-source languages, even though no fine-tuning is done on these languages. However, there is a clear gap between the performance of the source language and that of the non-source languages. This paper analyzes the fine-tuning process, discovers when the performance gap changes and identifies which network weights affect the overall performance most. Additionally, the paper seeks to answer to what extent the gap can be reduced by reducing forgetting. Based on the analysis results, a method named Fine-tuning slow and fast with four training policies is proposed to address these issues. Experimental results show the proposed method outperforms baselines by a clear margin.
Code (0)
등록된 구현이 없습니다.
Tasks
Cross-Lingual TransferLanguage ModelingLanguage ModellingSimilar Papers 제목 키워드 기반
DiTTO: A Feature Representation Imitation Approach for Improving Cross-Lingual Transfer
Zero-shot cross-lingual transfer is promising, however has been shown to be sub-optimal, with inferior transfer performance across low-resource languages. In this work, we envision languages as domains for improving zero…
Cross-Lingual TransferZero-Shot Cross-Lingual TransferAnalyzing the Evaluation of Cross-Lingual Knowledge Transfer in Multilingual Language Models
Recent advances in training multilingual language models on large datasets seem to have shown promising results in knowledge transfer across languages and achieve high performance on downstream tasks. However, we questio…
Transfer LearningAnalyzing the Effects of Reasoning Types on Cross-Lingual Transfer Performance
Multilingual language models achieve impressive zero-shot accuracies in many languages in complex tasks such as Natural Language Inference (NLI). Examples in NLI (and equivalent complex tasks) often pertain to various ty…
Cross-Lingual TransferNatural Language InferenceAnalyzing the Effect of Linguistic Similarity on Cross-Lingual Transfer: Tasks and Experimental Setups Matter
Cross-lingual transfer is a popular approach to increase the amount of training data for NLP tasks in a low-resource context. However, the best strategy to decide which cross-lingual data to include is unclear. Prior res…
Cross-Lingual TransferDependency ParsingPOSPOS Tagging+1Analyzing BERT Cross-lingual Transfer Capabilities in Continual Sequence Labeling
Knowledge transfer between neural language models is a widely used technique that has proven to improve performance in a multitude of natural language tasks, in particular with the recent rise of large pre-trained langua…
Continual LearningCross-Lingual Transfernamed-entity-recognitionNamed Entity Recognition+4