Modular-Relatedness for Continual Learning
In this paper, we propose a continual learning (CL) technique that is beneficial to sequential task learners by improving their retained accuracy and reducing catastrophic forgetting. The principal target of our approach is the automatic extraction of modular parts of the neural network and then estimating the relatedness between the tasks given these modular components. This technique is applicable to different families of CL methods such as regularization-based (e.g., the Elastic Weight Consolidation) or the rehearsal-based (e.g., the Gradient Episodic Memory) approaches where episodic memory is needed. Empirical results demonstrate remarkable performance gain (in terms of robustness to forgetting) for methods such as EWC and GEM based on our technique, especially when the memory budget is very limited.
Code (0)
등록된 구현이 없습니다.
Tasks
Continual LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Rehearsal-Free Modular and Compositional Continual Learning for Language Models
Continual learning aims at incrementally acquiring new knowledge while not forgetting existing knowledge. To overcome catastrophic forgetting, methods are either rehearsal-based, i.e., store data examples from previous t…
Continual LearningTransfer LearningPosition: Modular Memory is the Key to Continual Learning Agents
Foundation models have transformed machine learning through large-scale pretraining and increased test-time compute. Despite surpassing human performance in several domains, these models remain fundamentally limited in c…
Continual LearningDistillation-Guided Structural Transfer for Continual Learning Beyond Sparse Distributed Memory
Sparse neural systems are gaining traction for efficient continual learning due to their modularity and low interference. Architectures such as Sparse Distributed Memory Multi-Layer Perceptrons (SDMLP) construct task-spe…
Continual LearningMoDiCoL: A Modular Diagnostic Continual Learning Dataset for Robust Speech Recognition
Modern Automatic Speech Recognition (ASR) systems have made remarkable progress on standard benchmarks, yet performance gaps have emerged under real-world distribution shifts, caused by recording conditions, accents, spe…
Continual LearningSpeech RecognitionDimensionality Controls When Modularity Helps in Continual Learning
Compositional learning systems must balance plasticity, the ability to acquire new knowledge, with stability, the preservation of previously learned components, especially when tasks share structure and risk interference…
Continual Learning