Multi-task pre-finetuning for zero-shot cross lingual transfer
Building machine learning models for low resource languages is extremely challenging due to the lack of available training data (either un-annotated or annotated). To support such scenarios, zero-shot cross lingual transfer is used where the machine learning model is trained on a resource rich language and is directly tested on the resource poor language. In this paper, we present a technique which improves the performance of zero-shot cross lingual transfer. Our method performs multi-task pre-finetuning on a resource rich language using a multilingual pre-trained model. The pre-finetuned model is then tested in a zero-shot manner on the resource poor languages. We test the performance of our method on 8 languages and for two tasks, namely, Intent Classification (IC) & Named Entity Recognition (NER) using the MultiAtis++ dataset. The results showed that our method improves IC performance in 7 out of 8 languages and NER performance in 4 languages. Our method also leads to faster convergence during finetuning. The usage of pre-finetuning demonstrates a data efficient way for supporting new languages and geographies across the world.
Code (0)
등록된 구현이 없습니다.
Tasks
Cross-Lingual Transferintent-classificationIntent Classificationnamed-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)NERZero-Shot Cross-Lingual TransferSimilar Papers 제목 키워드 기반
The Benefits of Label-Description Training for Zero-Shot Text Classification
Pretrained language models have improved zero-shot text classification by allowing the transfer of semantic knowledge from the training data in order to classify among specific label sets in downstream tasks. We propose …
Classificationdomain classificationtext-classificationText Classification+3Small or Large? Zero-Shot or Finetuned? Guiding Language Model Choice for Specialized Applications in Healthcare
This study aims to guide language model selection by investigating: 1) the necessity of finetuning versus zero-shot usage, 2) the benefits of domain-adjacent versus generic pretrained models, 3) the value of further doma…
Language ModelingLanguage ModellingModel SelectionM$^2$PT: Multimodal Prompt Tuning for Zero-shot Instruction Learning
Multimodal Large Language Models (MLLMs) demonstrate remarkable performance across a wide range of domains, with increasing emphasis on enhancing their zero-shot generalization capabilities for unseen tasks across variou…
Zero-shot GeneralizationIs Prompt-Based Finetuning Always Better than Vanilla Finetuning? Insights from Cross-Lingual Language Understanding
Multilingual pretrained language models (MPLMs) have demonstrated substantial performance improvements in zero-shot cross-lingual transfer across various natural language understanding tasks by finetuning MPLMs on task-s…
Cross-Lingual TransferNatural Language InferenceNatural Language UnderstandingParaphrase Identification+3Crosslingual Generalization through Multitask Finetuning
Multitask prompted finetuning (MTF) has been shown to help large language models generalize to new tasks in a zero-shot setting, but so far explorations of MTF have focused on English data and models. We apply MTF to the…
Coreference ResolutionCross-Lingual TransferQuestion AnsweringSentence Completion+2