Online Continual Learning via the Meta-learning Update with Multi-scale Knowledge Distillation and Data Augmentation
Continual learning aims to rapidly and continually learn the current task from a sequence of tasks. Compared to other kinds of methods, the methods based on experience replay have shown great advantages to overcome catastrophic forgetting. One common limitation of this method is the data imbalance between the previous and current tasks, which would further aggravate forgetting. Moreover, how to effectively address the stability-plasticity dilemma in this setting is also an urgent problem to be solved. In this paper, we overcome these challenges by proposing a novel framework called Meta-learning update via Multi-scale Knowledge Distillation and Data Augmentation (MMKDDA). Specifically, we apply multiscale knowledge distillation to grasp the evolution of long-range and short-range spatial relationships at different feature levels to alleviate the problem of data imbalance. Besides, our method mixes the samples from the episodic memory and current task in the online continual training procedure, thus alleviating the side influence due to the change of probability distribution. Moreover, we optimize our model via the meta-learning update resorting to the number of tasks seen previously, which is helpful to keep a better balance between stability and plasticity. Finally, our experimental evaluation on four benchmark datasets shows the effectiveness of the proposed MMKDDA framework against other popular baselines, and ablation studies are also conducted to further analyze the role of each component in our framework.
Code (0)
등록된 구현이 없습니다.
Tasks
Continual LearningData AugmentationKnowledge DistillationMeta-LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
MANGO: Meta-Adaptive Network Gradient Optimization for Online Continual Learning
In Online Continual Learning (OCL), a neural network sequentially learns from a non-stationary data stream in a single-pass with access only to a limited memory replay buffer. This contrasts sharply with off-line continu…
Class Incremental LearningContinual LearningMeta-descent for Online, Continual Prediction
This paper investigates different vector step-size adaptation approaches for non-stationary online, continual prediction problems. Vanilla stochastic gradient descent can be considerably improved by scaling the update wi…
PredictionSecond-order methodsTime SeriesTime Series Analysis+1Look-ahead Meta Learning for Continual Learning
The continual learning problem involves training models with limited capacity to perform well on a set of an unknown number of sequentially arriving tasks. While meta-learning shows great potential for reducing interfer…
Continual LearningMeta-LearningLa-MAML: Look-ahead Meta Learning for Continual Learning
The continual learning problem involves training models with limited capacity to perform well on a set of an unknown number of sequentially arriving tasks. While meta-learning shows great potential for reducing interfere…
Continual LearningMeta-LearningLearning to Learn with Feedback and Local Plasticity
Interest in biologically inspired alternatives to backpropagation is driven by the desire to both advance connections between deep learning and neuroscience and address backpropagation's shortcomings on tasks such as onl…
Continual LearningMeta-Learning