Learning where to learn: Gradient sparsity in meta and continual learning
Finding neural network weights that generalize well from small datasets is difficult. A promising approach is to learn a weight initialization such that a small number of weight changes results in low generalization error. We show that this form of meta-learning can be improved by letting the learning algorithm decide which weights to change, i.e., by learning where to learn. We find that patterned sparsity emerges from this process, with the pattern of sparsity varying on a problem-by-problem basis. This selective sparsity results in better generalization and less interference in a range of few-shot and continual learning problems. Moreover, we find that sparse learning also emerges in a more expressive model where learning rates are meta-learned. Our results shed light on an ongoing debate on whether meta-learning can discover adaptable features and suggest that learning by sparse gradient descent is a powerful inductive bias for meta-learning systems.
Code (1)
Tasks
Continual LearningInductive BiasMeta-LearningSparse LearningSimilar Papers 제목 키워드 기반
Efficient Meta-Learning for Continual Learning with Taylor Expansion Approximation
Continual learning aims to alleviate catastrophic forgetting when handling consecutive tasks under non-stationary distributions. Gradient-based meta-learning algorithms have shown the capability to implicitly solve the t…
Computational EfficiencyContinual LearningMeta-LearningDual-Stage Invariant Continual Learning under Extreme Visual Sparsity
Continual learning seeks to maintain stable adaptation under non-stationary environments, yet this problem becomes particularly challenging in object detection, where most existing methods implicitly assume relatively ba…
Continual LearningObject DetectionEnabling Continual Learning in Neural Networks with Meta Learning
Catastrophic forgetting in neural networks is one of the most well-known problems in continual learning. Previous attempts on addressing the problem focus on preventing important weights from changing. Such methods often…
Continual LearningMeta-LearningTransfer LearningMANGO: Meta-Adaptive Network Gradient Optimization for Online Continual Learning
In Online Continual Learning (OCL), a neural network sequentially learns from a non-stationary data stream in a single-pass with access only to a limited memory replay buffer. This contrasts sharply with off-line continu…
Class Incremental LearningContinual LearningMeta Continual Learning via Dynamic Programming
Meta continual learning algorithms seek to train a model when faced with similar tasks observed in a sequential manner. Despite promising methodological advancements, there is a lack of theoretical frameworks that enable…
Continual Learning