Weight Friction: A Simple Method to Overcome Catastrophic Forgetting and Enable Continual Learning
In recent years, deep neural networks have found success in replicating human-level cognitive skills, yet they suffer from several major obstacles. One significant limitation is the inability to learn new tasks without forgetting previously learned tasks, a shortcoming known as catastrophic forgetting. In this research, we propose a simple method to overcome catastrophic forgetting and enable continual learning in neural networks. We draw inspiration from principles in neurology and physics to develop the concept of weight friction. Weight friction operates by a modification to the update rule in the gradient descent optimization method. It converges at a rate comparable to that of the stochastic gradient descent algorithm and can operate over multiple task domains. It performs comparably to current methods while offering improvements in computation and memory efficiency.
Code (0)
등록된 구현이 없습니다.
Tasks
Continual LearningFrictionSimilar Papers 제목 키워드 기반
Weight Averaging: A Simple Yet Effective Method to Overcome Catastrophic Forgetting in Automatic Speech Recognition
Adapting a trained Automatic Speech Recognition (ASR) model to new tasks results in catastrophic forgetting of old tasks, limiting the model's ability to learn continually and to be extended to new speakers, dialects, la…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Knowledge Distillationspeech-recognition+1Provable Continual Learning via Sketched Jacobian Approximations
An important problem in machine learning is the ability to learn tasks in a sequential manner. If trained with standard first-order methods most models forget previously learned tasks when trained on a new task, which is…
Continual LearningMitigating Catastrophic Forgetting in Long Short-Term Memory Networks
Continual learning on sequential data is critical for many machine learning (ML) deployments. Unfortunately, LSTM networks, which are commonly used to learn on sequential data, suffer from catastrophic forgetting and are…
Continual LearningDo You Remember? Overcoming Catastrophic Forgetting for Fake Audio Detection
Current fake audio detection algorithms have achieved promising performances on most datasets. However, their performance may be significantly degraded when dealing with audio of a different dataset. The orthogonal weigh…
Continual LearningEmotion RecognitionSpeech Emotion RecognitionNatural Way to Overcome the Catastrophic Forgetting in Neural Networks
Not so long ago, a method was discovered that successfully overcomes the catastrophic forgetting in neural networks. Although we know about the cases of using this method to preserve skills when adapting pre-trained netw…