Mitigating the Diminishing Effect of Elastic Weight Consolidation
Elastic weight consolidation (EWC, Kirkpatrick et al. 2017) is a promising approach to addressing catastrophic forgetting in sequential training. We find that the effect of EWC can diminish when fine-tuning large-scale pre-trained language models on different datasets. We present two simple objective functions to mitigate this problem by rescaling the components of EWC. Experiments on natural language inference and fact-checking tasks indicate that our methods require much smaller values for the trade-off parameters to achieve results comparable to EWC.
Code (0)
등록된 구현이 없습니다.
Tasks
Fact CheckingNatural Language InferenceSimilar Papers 제목 키워드 기반
EVCL: Elastic Variational Continual Learning with Weight Consolidation
Continual learning aims to allow models to learn new tasks without forgetting what has been learned before. This work introduces Elastic Variational Continual Learning with Weight Consolidation (EVCL), a novel hybrid mod…
Continual LearningIncremental LearningImproving Scheduled Sampling with Elastic Weight Consolidation for Neural Machine Translation
Despite strong performance in many sequence-to-sequence tasks, autoregressive models trained with maximum likelihood estimation suffer from exposure bias, i.e. the discrepancy between the ground-truth prefixes used durin…
Machine TranslationTranslationElastic Weight Consolidation (EWC): Nuts and Bolts
In this report, we present a theoretical support of the continual learning method \textbf{Elastic Weight Consolidation}, introduced in paper titled `Overcoming catastrophic forgetting in neural networks'. Being one of th…
Continual LearningElastic Weight Consolidation for Reduction of Catastrophic Forgetting in GPT-2
Neural networks are naturally prone to the effects of catastrophic forgetting during fine-tuning. Despite the extensive adoption of transformers, little research has been done to investigate the effects of catastrophic f…
On Quadratic Penalties in Elastic Weight Consolidation
Elastic weight consolidation (EWC, Kirkpatrick et al, 2017) is a novel algorithm designed to safeguard against catastrophic forgetting in neural networks. EWC can be seen as an approximation to Laplace propagation (Eskin…