Rotate your Networks: Better Weight Consolidation and Less Catastrophic Forgetting
In this paper we propose an approach to avoiding catastrophic forgetting in sequential task learning scenarios. Our technique is based on a network reparameterization that approximately diagonalizes the Fisher Information Matrix of the network parameters. This reparameterization takes the form of a factorized rotation of parameter space which, when used in conjunction with Elastic Weight Consolidation (which assumes a diagonal Fisher Information Matrix), leads to significantly better performance on lifelong learning of sequential tasks. Experimental results on the MNIST, CIFAR-100, CUB-200 and Stanford-40 datasets demonstrate that we significantly improve the results of standard elastic weight consolidation, and that we obtain competitive results when compared to other state-of-the-art in lifelong learning without forgetting.
Code (2)
Tasks
Lifelong learningSimilar Papers 제목 키워드 기반
Elastic weight consolidation for better bias inoculation
The biases present in training datasets have been shown to affect models for sentence pair classification tasks such as natural language inference (NLI) and fact verification. While fine-tuning models on additional data …
Fact VerificationGeneral ClassificationNatural Language InferenceSentence+1Overcoming catastrophic forgetting in neural networks
Catastrophic forgetting is the primary challenge that hinders continual learning, which refers to a neural network ability to sequentially learn multiple tasks while retaining previously acquired knowledge. Elastic Weigh…
Continual LearningL2 RegularizationLifelong learningRotate Your Character: Revisiting Video Diffusion Models for High-Quality 3D Character Generation
Generating high-quality 3D characters from single images remains a significant challenge in digital content creation, particularly due to complex body poses and self-occlusion. In this paper, we present RCM (Rotate your …
Novel View SynthesisVideo Generation3D GenerationEVCL: Elastic Variational Continual Learning with Weight Consolidation
Continual learning aims to allow models to learn new tasks without forgetting what has been learned before. This work introduces Elastic Variational Continual Learning with Weight Consolidation (EVCL), a novel hybrid mod…
Continual LearningIncremental LearningCatastrophic Forgetting in the Context of Model Updates
A large obstacle to deploying deep learning models in practice is the process of updating models post-deployment (ideally, frequently). Deep neural networks can cost many thousands of dollars to train. When new data come…
model