Continual Deep Learning by Functional Regularisation of Memorable Past
Continually learning new skills is important for intelligent systems, yet standard deep learning methods suffer from catastrophic forgetting of the past. Recent works address this with weight regularisation. Functional regularisation, although computationally expensive, is expected to perform better, but rarely does so in practice. In this paper, we fix this issue by using a new functional-regularisation approach that utilises a few memorable past examples crucial to avoid forgetting. By using a Gaussian Process formulation of deep networks, our approach enables training in weight-space while identifying both the memorable past and a functional prior. Our method achieves state-of-the-art performance on standard benchmarks and opens a new direction for life-long learning where regularisation and memory-based methods are naturally combined.
Code (1)
Tasks
Deep LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Monte Carlo Functional Regularisation for Continual Learning
Continual learning (CL) is crucial for the adaptation of neural network models to new environments. Although outperforming weight-space regularisation approaches, the functional regularisation-based CL methods suffer fro…
Continual LearningFunctional Regularisation for Continual Learning with Gaussian Processes
We introduce a framework for Continual Learning (CL) based on Bayesian inference over the function space rather than the parameters of a deep neural network. This method, referred to as functional regularisation for Cont…
Bayesian InferenceContinual LearningGaussian ProcessesSequential Bayesian InferenceAffect and Effect: Limitations of regularisation-based continual learning in EEG-based emotion classification
Generalisation to unseen subjects in EEG-based emotion classification remains a challenge due to high inter-and intra-subject variability. Continual learning (CL) poses a promising solution by learning from a sequence of…
Emotion ClassificationContinual LearningContinual learning via probabilistic exchangeable sequence modelling
Continual learning (CL) refers to the ability to continuously learn and accumulate new knowledge while retaining useful information from past experiences. Although numerous CL methods have been proposed in recent years, …
class-incremental learningClass Incremental LearningContinual LearningDecision Making+2A Framework of Meta Functional Learning for Regularising Knowledge Transfer
Machine learning classifiers' capability is largely dependent on the scale of available training data and limited by the model overfitting in data-scarce learning tasks. To address this problem, this work proposes a nove…
Cross-Domain Few-Shotcross-domain few-shot learningFew-Shot LearningMeta-Learning+1