Overcoming Catastrophic Forgetting During Domain Adaptation of Neural Machine Translation
Continued training is an effective method for domain adaptation in neural machine translation. However, in-domain gains from adaptation come at the expense of general-domain performance. In this work, we interpret the drop in general-domain performance as catastrophic forgetting of general-domain knowledge. To mitigate it, we adapt Elastic Weight Consolidation (EWC){---}a machine learning method for learning a new task without forgetting previous tasks. Our method retains the majority of general-domain performance lost in continued training without degrading in-domain performance, outperforming the previous state-of-the-art. We also explore the full range of general-domain performance available when some in-domain degradation is acceptable.
Code (0)
등록된 구현이 없습니다.
Tasks
BIG-bench Machine LearningDomain AdaptationMachine TranslationTranslationSimilar Papers 제목 키워드 기반
Domain adapted machine translation: What does catastrophic forgetting forget and why?
Neural Machine Translation (NMT) models can be specialized by domain adaptation, often involving fine-tuning on a dataset of interest. This process risks catastrophic forgetting: rapid loss of generic translation quality…
Domain AdaptationMachine TranslationNMTTranslationOvercoming Catastrophic Forgetting During Domain Adaptation of Seq2seq Language Generation
Seq2seq language generation models that are trained offline with multiple domains in a sequential fashion often suffer from catastrophic forgetting. Lifelong learning has been proposed to handle this problem. However, ex…
Domain AdaptationLifelong learningResponse GenerationText GenerationLifelong Domain Adaptive 3D Human Pose Estimation
3D Human Pose Estimation (3D HPE) is vital in various applications, from person re-identification and action recognition to virtual reality. However, the reliance on annotated 3D data collected in controlled environments…
Person Re-Identification3D Human Pose EstimationAction RecognitionDomain AdaptationBayesian Parameter-Efficient Fine-Tuning for Overcoming Catastrophic Forgetting
We are motivated primarily by the adaptation of text-to-speech synthesis models; however we argue that more generic parameter-efficient fine-tuning (PEFT) is an appropriate framework to do such adaptation. Nevertheless, …
Language ModelingLanguage Modellingparameter-efficient fine-tuningSpeech Synthesis+3Investigating Catastrophic Forgetting During Continual Training for Neural Machine Translation
Neural machine translation (NMT) models usually suffer from catastrophic forgetting during continual training where the models tend to gradually forget previously learned knowledge and swing to fit the newly added data w…
Domain AdaptationMachine TranslationNMTTranslation