Overcoming catastrophic forgetting in neural networks

2025-07-14 · Brandon Shuen Yi Loke, Filippo Quadri, Gabriel Vivanco, Maximilian Casagrande, Saúl Fenollosa

Catastrophic forgetting is the primary challenge that hinders continual learning, which refers to a neural network ability to sequentially learn multiple tasks while retaining previously acquired knowledge. Elastic Weight Consolidation, a regularization-based approach inspired by synaptic consolidation in biological neural systems, has been used to overcome this problem. In this study prior research is replicated and extended by evaluating EWC in supervised learning settings using the PermutedMNIST and RotatedMNIST benchmarks. Through systematic comparisons with L2 regularization and stochastic gradient descent (SGD) without regularization, we analyze how different approaches balance knowledge retention and adaptability. Our results confirm what was shown in previous research, showing that EWC significantly reduces forgetting compared to naive training while slightly compromising learning efficiency on new tasks. Moreover, we investigate the impact of dropout regularization and varying hyperparameters, offering insights into the generalization of EWC across diverse learning scenarios. These results underscore EWC's potential as a viable solution for lifelong learning in neural networks.

📄 PDF Abstract BibTeX arXiv:2507.10485

Code (0)

등록된 구현이 없습니다.

Tasks

Continual LearningL2 RegularizationLifelong learning

Methods 이 논문이 사용한 방법론

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

EWC The methon to overcome catastrophic forgetting in neural network while continual learning

Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting

2018-05-20 · NeurIPS 2018 12 · Hippolyt Ritter, Aleksandar Botev, David Barber

We introduce the Kronecker factored online Laplace approximation for overcoming catastrophic forgetting in neural networks. The method is grounded in a Bayesian online learning framework, where we recursively approximate…

Permuted-MNIST

ZeroFlow: Overcoming Catastrophic Forgetting is Easier than You Think

2025-01-02 · Tao Feng, Wei Li, Didi Zhu, Hangjie Yuan 외

Backpropagation provides a generalized configuration for overcoming catastrophic forgetting. Like, SGD and Adam are commonly used for weight updates in continual learning and continual pre-training. In practice, permissi…

Continual Learning

Attention-Based Structural-Plasticity

2019-03-02 · Soheil Kolouri, Nicholas Ketz, Xinyun Zou, Jeffrey Krichmar 외

Catastrophic forgetting/interference is a critical problem for lifelong learning machines, which impedes the agents from maintaining their previously learned knowledge while learning new tasks. Neural networks, in partic…

Lifelong learning Permuted-MNIST Split-MNIST

Overcoming Catastrophic Forgetting by XAI

2022-11-25 · Giang Nguyen

Explaining the behaviors of deep neural networks, usually considered as black boxes, is critical especially when they are now being adopted over diverse aspects of human life. Taking the advantages of interpretable machi…

Continual Learning Explainable Artificial Intelligence (XAI)Interpretable Machine Learning

Overcoming Catastrophic Forgetting in Graph Neural Networks

2020-12-10 · Huihui Liu, Yiding Yang, Xinchao Wang

Catastrophic forgetting refers to the tendency that a neural network "forgets" the previous learned knowledge upon learning new tasks. Prior methods have been focused on overcoming this problem on convolutional neural ne…