English Intermediate-Task Training Improves Zero-Shot Cross-Lingual Transfer Too
Intermediate-task training has been shown to substantially improve pretrained model performance on many language understanding tasks, at least in monolingual English settings. Here, we investigate whether English intermediate-task training is still helpful on non-English target tasks in a zero-shot cross-lingual setting. Using a set of 7 intermediate language understanding tasks, we evaluate intermediate-task transfer in a zero-shot cross-lingual setting on 9 target tasks from the XTREME benchmark. Intermediate-task training yields large improvements on the BUCC and Tatoeba tasks that use model representations directly without training, and moderate improvements on question-answering target tasks. Using SQuAD for intermediate training achieves the best results across target tasks, with an average improvement of 8.4 points on development sets. Selecting the best intermediate task model for each target task, we obtain a 6.1 point improvement over XLM-R Large on the XTREME benchmark, setting a new state of the art. Finally, we show that neither multi-task intermediate-task training nor continuing multilingual MLM during intermediate-task training offer significant improvements.
Code (0)
등록된 구현이 없습니다.
Tasks
Cross-Lingual TransferQuestion AnsweringXLM-RZero-Shot Cross-Lingual TransferMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
English Intermediate-Task Training Improves Zero-Shot Cross-Lingual Transfer Too
Intermediate-task training---fine-tuning a pretrained model on an intermediate task before fine-tuning again on the target task---often improves model performance substantially on language understanding tasks in monoling…
Cross-Lingual TransferHellaSwagQuestion AnsweringRetrieval+4Cross-lingual Intermediate Fine-tuning improves Dialogue State Tracking
Recent progress in task-oriented neural dialogue systems is largely focused on a handful of languages, as annotation of training data is tedious and expensive. Machine translation has been used to make systems multilingu…
Cross-Lingual TransferDialogue State TrackingMachine TranslationTransfer LearningSparse and Decorrelated Representations for Stable Zero-shot NMT
Using a single encoder and decoder for all directions and training with English-centric data is a popular scheme for multilingual NMT. However, zero-shot translation under this scheme is vulnerable to changes in training…
DecoderNMTTranslationCrosslingual Generalization through Multitask Finetuning
Multitask prompted finetuning (MTF) has been shown to help large language models generalize to new tasks in a zero-shot setting, but so far explorations of MTF have focused on English data and models. We apply MTF to the…
Coreference ResolutionCross-Lingual TransferQuestion AnsweringSentence Completion+2Evaluating the Effectiveness of Natural Language Inference for Hate Speech Detection in Languages with Limited Labeled Data
Most research on hate speech detection has focused on English where a sizeable amount of labeled training data is available. However, to expand hate speech detection into more languages, approaches that require minimal t…
Hate Speech DetectionNatural Language Inference