Class-Incremental Learning with Strong Pre-trained Models
Class-incremental learning (CIL) has been widely studied under the setting of starting from a small number of classes (base classes). Instead, we explore an understudied real-world setting of CIL that starts with a strong model pre-trained on a large number of base classes. We hypothesize that a strong base model can provide a good representation for novel classes and incremental learning can be done with small adaptations. We propose a 2-stage training scheme, i) feature augmentation -- cloning part of the backbone and fine-tuning it on the novel data, and ii) fusion -- combining the base and novel classifiers into a unified classifier. Experiments show that the proposed method significantly outperforms state-of-the-art CIL methods on the large-scale ImageNet dataset (e.g. +10% overall accuracy than the best). We also propose and analyze understudied practical CIL scenarios, such as base-novel overlap with distribution shift. Our proposed method is robust and generalizes to all analyzed CIL settings. Code is available at https://github.com/amazon-research/sp-cil.
Code (2)
Tasks
class-incremental learningClass Incremental LearningIncremental LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
On the Stability-Plasticity Dilemma of Class-Incremental Learning
A primary goal of class-incremental learning is to strike a balance between stability and plasticity, where models should be both stable enough to retain knowledge learned from previously seen classes, and plastic enough…
class-incremental learningClass Incremental LearningIncremental LearningRepresentation LearningIncremental Learning with Differentiable Architecture and Forgetting Search
As progress is made on training machine learning models on incrementally expanding classification tasks (i.e., incremental learning), a next step is to translate this progress to industry expectations. One technique miss…
Classificationimage-classificationImage ClassificationIncremental Learning+1PILOT: A Pre-Trained Model-Based Continual Learning Toolbox
While traditional machine learning can effectively tackle a wide range of problems, it primarily operates within a closed-world setting, which presents limitations when dealing with streaming data. As a solution, increme…
class-incremental learningClass Incremental LearningContinual LearningIncremental LearningInitial Classifier Weights Replay for Memoryless Class Incremental Learning
Incremental Learning (IL) is useful when artificial systems need to deal with streams of data and do not have access to all data at all times. The most challenging setting requires a constant complexity of the deep model…
Allclass-incremental learningClass Incremental LearningFairness+2Co-Transport for Class-Incremental Learning
Traditional learning systems are trained in closed-world for a fixed number of classes, and need pre-collected datasets in advance. However, new classes often emerge in real-world applications and should be learned incre…
class-incremental learningClass Incremental LearningIncremental Learning