Effective Fine-Tuning Methods for Cross-lingual Adaptation
Large scale multilingual pre-trained language models have shown promising results in zero- and few-shot cross-lingual tasks. However, recent studies have shown their lack of generalizability when the languages are structurally dissimilar. In this work, we propose a novel fine-tuning method based on co-training that aims to learn more generalized semantic equivalences as a complementary to multilingual language modeling using the unlabeled data in the target language. We also propose an adaption method based on contrastive learning to better capture the semantic relationship in the parallel data, when a few translation pairs are available. To show our method’s effectiveness, we conduct extensive experiments on cross-lingual inference and review classification tasks across various languages. We report significant gains compared to directly fine-tuning multilingual pre-trained models and other semi-supervised alternatives.
Code (0)
등록된 구현이 없습니다.
Tasks
Contrastive LearningLanguage ModelingLanguage ModellingTranslationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
On the Analysis of Cross-Lingual Prompt Tuning for Decoder-based Multilingual Model
An exciting advancement in the field of multilingual models is the emergence of autoregressive models with zero- and few-shot capabilities, a phenomenon widely reported in large-scale language models. To further improve …
DecoderNERparameter-efficient fine-tuningPOSIs Prompt-Based Finetuning Always Better than Vanilla Finetuning? Insights from Cross-Lingual Language Understanding
Multilingual pretrained language models (MPLMs) have demonstrated substantial performance improvements in zero-shot cross-lingual transfer across various natural language understanding tasks by finetuning MPLMs on task-s…
Cross-Lingual TransferNatural Language InferenceNatural Language UnderstandingParaphrase Identification+3Exploring Fine-tuning Techniques for Pre-trained Cross-lingual Models via Continual Learning
Recently, fine-tuning pre-trained language models (e.g., multilingual BERT) to downstream cross-lingual tasks has shown promising results. However, the fine-tuning process inevitably changes the parameters of the pre-tra…
Continual Learningnamed-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)+4Preserving Cross-Linguality of Pre-trained Models via Continual Learning
Recently, fine-tuning pre-trained language models (e.g., multilingual BERT) to downstream cross-lingual tasks has shown promising results. However, the fine-tuning process inevitably changes the parameters of the pre-tra…
Continual Learningnamed-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)+4Self-Translate-Train: Enhancing Cross-Lingual Transfer of Large Language Models via Inherent Capability
Zero-shot cross-lingual transfer by fine-tuning multilingual pretrained models shows promise for low-resource languages, but often suffers from misalignment of internal representations between languages. We hypothesize t…
Cross-Lingual TransferLanguage ModellingLarge Language ModelTranslation+1