Decision Boundary-aware Knowledge Consolidation Generates Better Instance-Incremental Learner
Instance-incremental learning (IIL) focuses on learning continually with data of the same classes. Compared to class-incremental learning (CIL), the IIL is seldom explored because IIL suffers less from catastrophic forgetting (CF). However, besides retaining knowledge, in real-world deployment scenarios where the class space is always predefined, continual and cost-effective model promotion with the potential unavailability of previous data is a more essential demand. Therefore, we first define a new and more practical IIL setting as promoting the model's performance besides resisting CF with only new observations. Two issues have to be tackled in the new IIL setting: 1) the notorious catastrophic forgetting because of no access to old data, and 2) broadening the existing decision boundary to new observations because of concept drift. To tackle these problems, our key insight is to moderately broaden the decision boundary to fail cases while retain old boundary. Hence, we propose a novel decision boundary-aware distillation method with consolidating knowledge to teacher to ease the student learning new knowledge. We also establish the benchmarks on existing datasets Cifar-100 and ImageNet. Notably, extensive experiments demonstrate that the teacher model can be a better incremental learner than the student model, which overturns previous knowledge distillation-based methods treating student as the main role.
Code (0)
등록된 구현이 없습니다.
Tasks
class-incremental learningClass Incremental LearningIncremental LearningKnowledge DistillationSimilar Papers 제목 키워드 기반
Boundary-Aware Uncertainty for Feature Attribution Explainers
Post-hoc explanation methods have become a critical tool for understanding black-box classifiers in high-stakes applications. However, high-performing classifiers are often highly nonlinear and can exhibit complex behavi…
DESIRE: Dynamic Knowledge Consolidation for Rehearsal-Free Continual Learning
Continual learning aims to equip models with the ability to retain previously learned knowledge like a human. Recent work incorporating Parameter-Efficient Fine-Tuning has revitalized the field by introducing lightweight…
Continual Learningparameter-efficient fine-tuningGRSDet: Learning to Generate Local Reverse Samples for Few-shot Object Detection
Few-shot object detection (FSOD) aims to achieve object detection only using a few novel class training data. Most of the existing methods usually adopt a transfer-learning strategy to construct the novel class distribut…
Few-Shot Object Detectionobject-detectionObject DetectionTransfer LearningKnowledge Distillation with Adversarial Samples Supporting Decision Boundary
Many recent works on knowledge distillation have provided ways to transfer the knowledge of a trained network for improving the learning process of a new one, but finding a good technique for knowledge distillation is st…
Adversarial AttackKnowledge DistillationDSKC: Domain Style Modeling with Adaptive Knowledge Consolidation for Exemplar-free Lifelong Person Re-Identification
Lifelong Person Re-identification (LReID) aims to continuously match individuals across camera views from sequential data streams. Existing LReID methods often ignore domain-specific style awareness and unified knowledge…
Person Re-Identification