MultiFiT: Efficient Multi-lingual Language Model Fine-tuning
Pretrained language models are promising particularly for low-resource languages as they only require unlabelled data. However, training existing models requires huge amounts of compute, while pretrained cross-lingual models often underperform on low-resource languages. We propose Multi-lingual language model Fine-Tuning (MultiFiT) to enable practitioners to train and fine-tune language models efficiently in their own language. In addition, we propose a zero-shot method using an existing pretrained cross-lingual model. We evaluate our methods on two widely used cross-lingual classification datasets where they outperform models pretrained on orders of magnitude more data and compute. We release all models and code.
Code (4)
Tasks
Cross-Lingual Document ClassificationDocument ClassificationLanguage ModelingLanguage ModellingmodelZero-shot Cross-Lingual Document ClassificationSimilar Papers 제목 키워드 기반
Discussion of `Multiscale Fisher's Independence Test for Multivariate Dependence'
We discuss how MultiFIT, the Multiscale Fisher's Independence Test for Multivariate Dependence proposed by Gorsky and Ma (2022), compares to existing linear-time kernel tests based on the Hilbert-Schmidt independence cri…
Multilingual Translation with Extensible Multilingual Pretraining and Finetuning
Recent work demonstrates the potential of multilingual pretraining of creating one model that can be used for various tasks in different languages. Previous work in multilingual pretraining has demonstrated that machine …
Machine TranslationTranslationOn the Analysis of Cross-Lingual Prompt Tuning for Decoder-based Multilingual Model
An exciting advancement in the field of multilingual models is the emergence of autoregressive models with zero- and few-shot capabilities, a phenomenon widely reported in large-scale language models. To further improve …
DecoderNERparameter-efficient fine-tuningPOSIs Prompt-Based Finetuning Always Better than Vanilla Finetuning? Insights from Cross-Lingual Language Understanding
Multilingual pretrained language models (MPLMs) have demonstrated substantial performance improvements in zero-shot cross-lingual transfer across various natural language understanding tasks by finetuning MPLMs on task-s…
Cross-Lingual TransferNatural Language InferenceNatural Language UnderstandingParaphrase Identification+3Prompt-Tuning Can Be Much Better Than Fine-Tuning on Cross-lingual Understanding With Multilingual Language Models
Pre-trained multilingual language models show significant performance gains for zero-shot cross-lingual model transfer on a wide range of natural language understanding (NLU) tasks. Previously, for zero-shot cross-lingua…
Cross-Lingual TransferNatural Language UnderstandingQuestion AnsweringSentence+1