InstructAlign: High-and-Low Resource Language Alignment via Continual Crosslingual Instruction Tuning
Large language models (LLMs) that are tuned with instructions have demonstrated remarkable capabilities in various tasks and languages. However, their ability to generalize to underrepresented languages is limited due to the scarcity of available data. Additionally, directly adapting new languages to instruction-tuned LLMs can result in catastrophic forgetting, which leads to the loss of multitasking ability. To address this issue, we propose InstructAlign which uses continual crosslingual instruction tuning to enable LLMs to align new unseen languages with previously learned high-resource languages. Our results demonstrate the effectiveness of InstructAlign in enabling the model to understand low-resource languages with limited parallel data while preventing catastrophic forgetting. Our work contributes to the advancement of language adaptation methods, particularly for adapting instruction-tuned LLMs to underrepresented languages. Our code is released on https://github.com/HLTCHKUST/InstructAlign
Code (1)
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Pipeline Analysis for Developing Instruct LLMs in Low-Resource Languages: A Case Study on Basque
Large language models (LLMs) are typically optimized for resource-rich languages like English, exacerbating the gap between high-resource and underrepresented languages. This work presents a detailed analysis of strategi…
Instruction FollowingNatural Language UnderstandingRevisiting Replay and Gradient Alignment for Continual Pre-Training of Large Language Models
Training large language models (LLMs) typically involves pre-training on massive corpora, only to restart the process entirely when new data becomes available. A more efficient and resource-conserving approach would be c…
Continual LearningContinual Learning for VLMs: A Survey and Taxonomy Beyond Forgetting
Vision-language models (VLMs), spanning predictive architectures to generative Multimodal Large Language Models (MLLMs), have revolutionized artificial intelligence through powerful cross-modal alignment and zero-shot ge…
Compositional Zero-Shot LearningZero-shot GeneralizationContinual LearningMending the Holes: Mitigating Reward Hacking in Reinforcement Learning for Multilingual Translation
Large Language Models (LLMs) have demonstrated remarkable capability in machine translation on high-resource language pairs, yet their performance on low-resource translation still lags behind. Existing post-training met…
Reinforcement LearningMachine TranslationWord AlignmentInstructionCP: A fast approach to transfer Large Language Models into target language
The rapid development of large language models (LLMs) in recent years has largely focused on English, resulting in models that respond exclusively in English. To adapt these models to other languages, continual pre-train…
Instruction Following