Continual Learners are Incremental Model Generalizers
Motivated by the efficiency and rapid convergence of pre-trained models for solving downstream tasks, this paper extensively studies the impact of Continual Learning (CL) models as pre-trainers. In both supervised and unsupervised CL, we find that the transfer quality of the representation often increases gradually without noticeable degradation in fine-tuning performance. This is because CL models can learn improved task-general features when easily forgetting task-specific knowledge. Based on this observation, we suggest a new unsupervised CL framework with masked modeling, which aims to capture fluent task-generic representation during training. Furthermore, we propose a new fine-tuning scheme, GLobal Attention Discretization (GLAD), that preserves rich task-generic representation during solving downstream tasks. The model fine-tuned with GLAD achieves competitive performance and can also be used as a good pre-trained model itself. We believe this paper breaks the barriers between pre-training and fine-tuning steps and leads to a sustainable learning framework in which the continual learner incrementally improves model generalization, yielding better transfer to unseen tasks.
Code (0)
등록된 구현이 없습니다.
Tasks
Continual LearningmodelSimilar Papers 제목 키워드 기반
What Can AutoML Do For Continual Learning?
This position paper outlines the potential of AutoML for incremental (continual) learning to encourage more research in this direction. Incremental learning involves incorporating new data from a stream of tasks and dist…
AutoMLContinual LearningIncremental LearningNoise-Tolerant Coreset-Based Class Incremental Continual Learning
Many applications of computer vision require the ability to adapt to novel data distributions after deployment. Adaptation requires algorithms capable of continual learning (CL). Continual learners must be plastic to ada…
class-incremental learningClass Incremental LearningContinual LearningIncremental LearningCoSCL: Cooperation of Small Continual Learners is Stronger than a Big One
Continual learning requires incremental compatibility with a sequence of tasks. However, the design of model architecture remains an open question: In general, learning all tasks with a shared set of parameters suffers f…
Continual LearningOpen-Ended Question AnsweringAttack On Prompt: Backdoor Attack in Prompt-Based Continual Learning
Prompt-based approaches offer a cutting-edge solution to data privacy issues in continual learning, particularly in scenarios involving multiple data suppliers where long-term storage of private user data is prohibited. …
Backdoor AttackContinual LearningIncremental LearningClass-Incremental Learning Using Generative Experience Replay Based on Time-aware Regularization
Learning new tasks accumulatively without forgetting remains a critical challenge in continual learning. Generative experience replay addresses this challenge by synthesizing pseudo-data points for past learned tasks and…
class-incremental learningClass Incremental LearningContinual LearningIncremental Learning