Continual Learning Using Task Conditional Neural Networks
Conventional deep learning models have limited capacity in learning multiple tasks sequentially. The issue of forgetting the previously learned tasks in continual learning is known as catastrophic forgetting or interference. When the input data or the goal of learning changes, a continual model will learn and adapt to the new status. However, the model will not remember or recognise any revisits to the previous states. This causes performance reduction and re-training curves in dealing with periodic or irregularly reoccurring changes in the data or goals. Dynamic approaches, which assign new neuron resources to the upcoming tasks, are introduced to address this issue. However, most of the dynamic methods need task information about the upcoming tasks during the inference phase to activate the corresponding neurons. To address this issue, we introduce Task Conditional Neural Network which allows the model to identify the task information automatically. The proposed model can continually learn and embed new tasks into the model without losing the information about previously learned tasks. We evaluate the proposed model combined with the mixture of experts approach on the MNIST and CIFAR100 datasets and show how it significantly improves the continual learning process without requiring task information in advance.
Code (0)
등록된 구현이 없습니다.
Tasks
Continual LearningMixture-of-ExpertsSimilar Papers 제목 키워드 기반
Marginal Replay vs Conditional Replay for Continual Learning
We present a new replay-based method of continual classification learning that we term "conditional replay" which generates samples and labels together by sampling from a distribution conditioned on the class. We compare…
ClassificationContinual LearningGeneral ClassificationContinual Learning Using Multi-view Task Conditional Neural Networks
Conventional deep learning models have limited capacity in learning multiple tasks sequentially. The issue of forgetting the previously learned tasks in continual learning is known as catastrophic forgetting or interfere…
Continual LearningOnline Continual Learning Under Domain Shift
Existing continual learning benchmarks often assume each task's training and test data are from the same distribution, which may not hold in practice. Towards making continual learning practical, in this paper, we introd…
Continual LearningEXTENDING CONDITIONAL CONVOLUTION STRUCTURES FOR ENHANCING MULTITASKING CONTINUAL LEARNING
Conditional operations have received much attention in recent deep learning studies to facilitate the prediction accuracy of a model. A recent advance toward this direction is the conditional parametric convolutions (Con…
Computational EfficiencyContinual LearningMulti-Task LearningConditional Input Gated Low-Rank Perturbations for Continual Learning
We address the problem of learning convolution neural networks (CNN) in the continual setting when tasks arrive sequentially, and only the data of the current task is available. In this setting CNNs are prone to reduce t…
Continual LearningSplit-MNIST