What Causes Knowledge Loss in Multilingual Language Models?
Cross-lingual transfer in natural language processing (NLP) models enhances multilingual performance by leveraging shared linguistic knowledge. However, traditional methods that process all data simultaneously often fail to mimic real-world scenarios, leading to challenges like catastrophic forgetting, where fine-tuning on new tasks degrades performance on previously learned ones. Our study explores this issue in multilingual contexts, focusing on linguistic differences affecting representational learning rather than just model parameters. We experiment with 52 languages using LoRA adapters of varying ranks to evaluate non-shared, partially shared, and fully shared parameters. Our aim is to see if parameter sharing through adapters can mitigate forgetting while preserving prior knowledge. We find that languages using non-Latin scripts are more susceptible to catastrophic forgetting, whereas those written in Latin script facilitate more effective cross-lingual transfer.
Code (0)
등록된 구현이 없습니다.
Tasks
Cross-Lingual TransferSimilar Papers 제목 키워드 기반
To What Degree Can Language Borders Be Blurred In BERT-based Multilingual Spoken Language Understanding?
This paper addresses the question as to what degree a BERT-based multilingual Spoken Language Understanding (SLU) model can transfer knowledge across languages. Through experiments we will show that, although it works su…
Spoken Language UnderstandingAnalyzing the Evaluation of Cross-Lingual Knowledge Transfer in Multilingual Language Models
Recent advances in training multilingual language models on large datasets seem to have shown promising results in knowledge transfer across languages and achieve high performance on downstream tasks. However, we questio…
Transfer LearningRetrieval-augmented Multilingual Knowledge Editing
Knowledge represented in Large Language Models (LLMs) is quite often incorrect and can also become obsolete over time. Updating knowledge via fine-tuning is computationally resource-hungry and not reliable, and so knowle…
knowledge editingRetrievalLanguage Bias under Conflicting Information in Multilingual LLMs
Large Language Models (LLMs) have been shown to contain biases in the process of integrating conflicting information when answering questions. Here we ask whether such biases also exist with respect to which language is …
XLM-K: Improving Cross-Lingual Language Model Pre-training with Multilingual Knowledge
Cross-lingual pre-training has achieved great successes using monolingual and bilingual plain text corpora. However, most pre-trained models neglect multilingual knowledge, which is language agnostic but comprises abunda…
Language ModelingLanguage ModellingNER