Self-Supervised Models are Continual Learners
Self-supervised models have been shown to produce comparable or better visual representations than their supervised counterparts when trained offline on unlabeled data at scale. However, their efficacy is catastrophically reduced in a Continual Learning (CL) scenario where data is presented to the model sequentially. In this paper, we show that self-supervised loss functions can be seamlessly converted into distillation mechanisms for CL by adding a predictor network that maps the current state of the representations to their past state. This enables us to devise a framework for Continual self-supervised visual representation Learning that (i) significantly improves the quality of the learned representations, (ii) is compatible with several state-of-the-art self-supervised objectives, and (iii) needs little to no hyperparameter tuning. We demonstrate the effectiveness of our approach empirically by training six popular self-supervised models in various CL settings.
Code (1)
Tasks
Continual LearningRepresentation LearningSimilar Papers 제목 키워드 기반
Domain-Aware Augmentations for Unsupervised Online General Continual Learning
Continual Learning has been challenging, especially when dealing with unsupervised scenarios such as Unsupervised Online General Continual Learning (UOGCL), where the learning agent has no prior knowledge of class bounda…
Continual LearningContrastive LearningSelf-Supervised Continual Graph Learning in Adaptive Riemannian Spaces
Continual graph learning routinely finds its role in a variety of real-world applications where the graph data with different tasks come sequentially. Despite the success of prior works, it still faces great challenges. …
Graph LearningMasked Autoencoders are Parameter-Efficient Federated Continual Learners
Federated learning is a specific distributed learning paradigm in which a central server aggregates updates from multiple clients' local models, thereby enabling the server to learn without requiring clients to upload th…
Continual LearningFederated LearningImage ReconstructionContinual Learners are Incremental Model Generalizers
Motivated by the efficiency and rapid convergence of pre-trained models for solving downstream tasks, this paper extensively studies the impact of Continual Learning (CL) models as pre-trainers. In both supervised and un…
Continual LearningmodelContrastive Continual Learning with Feature Propagation
Classical machine learners are designed only to tackle one task without capability of adopting new emerging tasks or classes whereas such capacity is more practical and human-like in the real world. To address this short…
Continual LearningRepresentation Learning