Natural Way to Overcome the Catastrophic Forgetting in Neural Networks
Not so long ago, a method was discovered that successfully overcomes the catastrophic forgetting in neural networks. Although we know about the cases of using this method to preserve skills when adapting pre-trained networks to particular tasks, it has not obtained widespread distribution yet. In this paper, we would like to propose an alternative method of overcoming catastrophic forgetting based on the total absolute signal passing through each connection in the network. This method has a simple implementation and seems to us essentially close to the processes occurring in the brain of animals to preserve previously learned skills during subsequent learning. We hope that the ease of implementation of this method will serve its wide application.
Code (1)
Similar Papers 제목 키워드 기반
Addressing Catastrophic Forgetting in Few-Shot Problems
Neural networks are known to suffer from catastrophic forgetting when trained on sequential datasets. While there have been numerous attempts to solve this problem in large-scale supervised classification, little has bee…
ClassificationGeneral ClassificationMeta-LearningVariational InferenceMitigating Catastrophic Forgetting in Long Short-Term Memory Networks
Continual learning on sequential data is critical for many machine learning (ML) deployments. Unfortunately, LSTM networks, which are commonly used to learn on sequential data, suffer from catastrophic forgetting and are…
Continual LearningQuantum Continual Learning Overcoming Catastrophic Forgetting
Catastrophic forgetting describes the fact that machine learning models will likely forget the knowledge of previously learned tasks after the learning process of a new one. It is a vital problem in the continual learnin…
BIG-bench Machine LearningContinual LearningIncremental LearningQuantum Machine LearningWeight Friction: A Simple Method to Overcome Catastrophic Forgetting and Enable Continual Learning
In recent years, deep neural networks have found success in replicating human-level cognitive skills, yet they suffer from several major obstacles. One significant limitation is the inability to learn new tasks without f…
Continual LearningFrictionSelfAug: Mitigating Catastrophic Forgetting in Retrieval-Augmented Generation via Distribution Self-Alignment
Recent advancements in large language models (LLMs) have revolutionized natural language processing through their remarkable capabilities in understanding and executing diverse tasks. While supervised fine-tuning, partic…