Exploring Pretraining via Active Forgetting for Improving Cross Lingual Transfer for Decoder Language Models
Large Language Models (LLMs) demonstrate exceptional capabilities in a multitude of NLP tasks. However, the efficacy of such models to languages other than English is often limited. Prior works have shown that encoder-only models such as BERT or XLM-RoBERTa show impressive cross lingual transfer of their capabilities from English to other languages. In this work, we propose a pretraining strategy that uses active forgetting to achieve similar cross lingual transfer in decoder-only LLMs. We show that LLMs pretrained with active forgetting are highly effective when adapting to new and unseen languages. Through extensive experimentation, we find that LLMs pretrained with active forgetting are able to learn better multilingual representations which translates to better performance in many downstream tasks.
Code (0)
등록된 구현이 없습니다.
Tasks
Cross-Lingual TransferDecoderMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
MiLorE-SSL: Scaling Multilingual Capabilities in Self-Supervised Models without Forgetting
Self-supervised learning (SSL) has greatly advanced speech representation learning, but multilingual SSL models remain constrained to languages encountered during pretraining. Retraining from scratch to incorporate new l…
Self-Supervised LearningRepresentation LearningMeasuring Catastrophic Forgetting in Cross-Lingual Transfer Paradigms: Exploring Tuning Strategies
The cross-lingual transfer is a promising technique to solve tasks in less-resourced languages. In this empirical study, we compare two fine-tuning approaches combined with zero-shot and full-shot learning approaches for…
Cross-Lingual TransferHate Speech DetectionWhen Does Language Transfer Help? Sequential Fine-Tuning for Cross-Lingual Euphemism Detection
Euphemisms are culturally variable and often ambiguous, posing challenges for language models, especially in low-resource settings. This paper investigates how cross-lingual transfer via sequential fine-tuning affects eu…
Cross-Lingual TransferParameter Alignment Mitigates Catastrophic Forgetting in Multilingual Expert Language Models
While continual pretraining~(CPT) is a practical way to extend large language models to new languages, naïve finetuning on targeted data erodes existing capabilities through catastrophic forgetting. Organizing training a…
Reading ComprehensionContinual PretrainingLanguage AcquisitionGeneral KnowledgeExploring Cross-Lingual Guidance in Abstractive Summarization
Cross-lingual guidance (CLG) as an augmentation method is often applied in cross-lingual summarization (CLS) to improve its performance. In this paper, we empirically study how cross-lingual information of different qual…
Abstractive Text Summarization