English Intermediate-Task Training Improves Zero-Shot Cross-Lingual Transfer Too
Intermediate-task training---fine-tuning a pretrained model on an intermediate task before fine-tuning again on the target task---often improves model performance substantially on language understanding tasks in monolingual English settings. We investigate whether English intermediate-task training is still helpful on non-English target tasks. Using nine intermediate language-understanding tasks, we evaluate intermediate-task transfer in a zero-shot cross-lingual setting on the XTREME benchmark. We see large improvements from intermediate training on the BUCC and Tatoeba sentence retrieval tasks and moderate improvements on question-answering target tasks. MNLI, SQuAD and HellaSwag achieve the best overall results as intermediate tasks, while multi-task intermediate offers small additional improvements. Using our best intermediate-task models for each target task, we obtain a 5.4 point improvement over XLM-R Large on the XTREME benchmark, setting the state of the art as of June 2020. We also investigate continuing multilingual MLM during intermediate-task training and using machine-translated intermediate-task data, but neither consistently outperforms simply performing English intermediate-task training.
Code (0)
등록된 구현이 없습니다.
Tasks
Cross-Lingual TransferHellaSwagQuestion AnsweringRetrievalSentenceSentence RetrievalXLM-RZero-Shot Cross-Lingual TransferMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
English Intermediate-Task Training Improves Zero-Shot Cross-Lingual Transfer Too
Intermediate-task training has been shown to substantially improve pretrained model performance on many language understanding tasks, at least in monolingual English settings. Here, we investigate whether English interme…
Cross-Lingual TransferQuestion AnsweringXLM-RZero-Shot Cross-Lingual TransferCross-lingual Intermediate Fine-tuning improves Dialogue State Tracking
Recent progress in task-oriented neural dialogue systems is largely focused on a handful of languages, as annotation of training data is tedious and expensive. Machine translation has been used to make systems multilingu…
Cross-Lingual TransferDialogue State TrackingMachine TranslationTransfer LearningSparse and Decorrelated Representations for Stable Zero-shot NMT
Using a single encoder and decoder for all directions and training with English-centric data is a popular scheme for multilingual NMT. However, zero-shot translation under this scheme is vulnerable to changes in training…
DecoderNMTTranslationCrosslingual Generalization through Multitask Finetuning
Multitask prompted finetuning (MTF) has been shown to help large language models generalize to new tasks in a zero-shot setting, but so far explorations of MTF have focused on English data and models. We apply MTF to the…
Coreference ResolutionCross-Lingual TransferQuestion AnsweringSentence Completion+2Evaluating the Effectiveness of Natural Language Inference for Hate Speech Detection in Languages with Limited Labeled Data
Most research on hate speech detection has focused on English where a sizeable amount of labeled training data is available. However, to expand hate speech detection into more languages, approaches that require minimal t…
Hate Speech DetectionNatural Language Inference