Efficient Continual Learning in Neural Networks with Embedding Regularization
Continual learning of deep neural networks is a key requirement for scaling them up to more complex applicative scenarios and for achieving real lifelong learning of these architectures. Previous approaches to the problem have considered either the progressive increase in the size of the networks, or have tried to regularize the network behavior to equalize it with respect to previously observed tasks. In the latter case, it is essential to understand what type of information best represents this past behavior. Common techniques include regularizing the past outputs, gradients, or individual weights. In this work, we propose a new, relatively simple and efficient method to perform continual learning by regularizing instead the network internal embeddings. To make the approach scalable, we also propose a dynamic sampling strategy to reduce the memory footprint of the required external storage. We show that our method performs favorably with respect to state-of-the-art approaches in the literature, while requiring significantly less space in memory and computational time. In addition, inspired inspired by to recent works, we evaluate the impact of selecting a more flexible model for the activation functions inside the network, evaluating the impact of catastrophic forgetting on the activation functions themselves.
Code (1)
Tasks
Continual LearningLifelong learningSimilar Papers 제목 키워드 기반
Continual Few-shot Relation Learning via Embedding Space Regularization and Data Augmentation
Existing continual relation learning (CRL) methods rely on plenty of labeled training data for learning a new task, which can be hard to acquire in real scenario as getting large and representative labeled data is often …
Data AugmentationRelationContinual Few-shot Relation Learning via Embedding Space Regularization and Data Augmentation
Existing continual relation learning (CRL) methods rely on plenty of labeled training data for learning a new task, which can be hard to acquire in real scenario as getting large and representative labeled data is often …
Data AugmentationRelationTowards Exemplar-Free Continual Learning in Vision Transformers: an Account of Attention, Functional and Weight Regularization
In this paper, we investigate the continual learning of Vision Transformers (ViT) for the challenging exemplar-free scenario, with special focus on how to efficiently distill the knowledge of its crucial self-attention m…
Continual LearningExemplar-FreeElastic Weight Consolidation for Knowledge Graph Continual Learning: An Empirical Evaluation
Knowledge graphs (KGs) require continual updates as new information emerges, but neural embedding models suffer from catastrophic forgetting when learning new tasks sequentially. We evaluate Elastic Weight Consolidation …
Continual LearningKnowledge GraphsLink PredictionMatched Excess-Outranker Regularization for Candidate-Set Interference in Continual Knowledge Graph Embedding
Continual knowledge graph embedding updates entity and relation representations as a graph grows. Existing methods primarily address catastrophic forgetting, but entity admission also changes the candidate universe of ev…
Knowledge Graph Embedding