Few-Shot Regularization to Tackle Catastrophic Forgetting in Multilingual Machine Translation
Increasing the number of tasks supported by a machine learning model without forgetting previously learned tasks is the goal of any lifelong learning system. In this work, we study how to mitigate the effects of the catastrophic forgetting problem to sequentially train a multilingual neural machine translation model using minimal past information. First, we describe the catastrophic forgetting phenomenon as a function of the number of tasks learned (language pairs) and the ratios of past data used during the learning of the new task. Next, we explore the importance of applying oversampling strategies for scenarios where only minimal amounts of past data are available. Finally, we derive a new loss function that minimizes the forgetting of previously learned tasks by actively re-weighting past samples and penalizing weights that deviate too much from the original model. Our work suggests that by using minimal amounts of past data and a simple regularization function, we can significantly mitigate the effects of the catastrophic forgetting phenomenon without increasing the computational costs.
Code (0)
등록된 구현이 없습니다.
Tasks
Lifelong learningMachine TranslationTranslationSimilar Papers 제목 키워드 기반
Addressing Catastrophic Forgetting in Few-Shot Problems
Neural networks are known to suffer from catastrophic forgetting when trained on sequential datasets. While there have been numerous attempts to solve this problem in large-scale supervised classification, little has bee…
ClassificationGeneral ClassificationMeta-LearningVariational InferenceActivity Regularization for Continual Learning
While deep neural networks have achieved remarkable successes, they suffer the well-known catastrophic forgetting issue when switching from existing tasks to tackle a new one. In this paper, we study continual learning w…
Continual LearningMulti-Task LearningIncremental few-shot learning via vector quantization in deep embedded space
The capability of incrementally learning new tasks without forgetting old ones is a challenging problem due to catastrophic forgetting. This challenge becomes greater when novel tasks contain very few labelled training s…
class-incremental learningClass Incremental LearningFew-Shot Class-Incremental LearningFew-Shot Learning+3Parameter Alignment Mitigates Catastrophic Forgetting in Multilingual Expert Language Models
While continual pretraining~(CPT) is a practical way to extend large language models to new languages, naïve finetuning on targeted data erodes existing capabilities through catastrophic forgetting. Organizing training a…
Reading ComprehensionContinual PretrainingLanguage AcquisitionGeneral KnowledgeBayesian Online Meta-Learning
Neural networks are known to suffer from catastrophic forgetting when trained on sequential datasets. While there have been numerous attempts to solve this problem for large-scale supervised classification, little has be…
ClassificationMeta-LearningVariational Inference