Class Incremental Learning with Multi-Teacher Distillation
Distillation strategies are currently the primary approaches for mitigating forgetting in class incremental learning (CIL). Existing methods generally inherit previous knowledge from a single teacher. However teachers with different mechanisms are talented at different tasks and inheriting diverse knowledge from them can enhance compatibility with new knowledge. In this paper we propose the MTD method to find multiple diverse teachers for CIL. Specifically we adopt weight permutation feature perturbation and diversity regularization techniques to ensure diverse mechanisms in teachers. To reduce time and memory consumption each teacher is represented as a small branch in the model. We adapt existing CIL distillation strategies with MTD and extensive experiments on CIFAR-100 ImageNet-100 and ImageNet-1000 show significant performance improvement. Our code is available at https://github.com/HaitaoWen/CLearning.
Code (0)
등록된 구현이 없습니다.
Tasks
class-incremental learningClass Incremental LearningDiversityIncremental LearningSimilar Papers 제목 키워드 기반
Multi-Teacher Knowledge Distillation for Incremental Implicitly-Refined Classification
Incremental learning methods can learn new classes continually by distilling knowledge from the last model (as a teacher model) to the current model (as a student model) in the sequentially learning process. However, the…
ClassificationIncremental LearningKnowledge DistillationDual-Teacher Class-Incremental Learning With Data-Free Generative Replay
This paper proposes two novel knowledge transfer techniques for class-incremental learning (CIL). First, we propose data-free generative replay (DF-GR) to mitigate catastrophic forgetting in CIL by using synthetic sample…
class-incremental learningClass Incremental LearningIncremental LearningKnowledge Distillation+1Adapt Your Teacher: Improving Knowledge Distillation for Exemplar-free Continual Learning
In this work, we investigate exemplar-free class incremental learning (CIL) with knowledge distillation (KD) as a regularization strategy, aiming to prevent forgetting. KD-based methods are successfully used in CIL, but …
class-incremental learningClass Incremental LearningContinual LearningExemplar-Free+3Re-examining Distillation For Continual Object Detection
Training models continually to detect and classify objects, from new classes and new domains, remains an open problem. In this work, we conduct a thorough analysis of why and how object detection models forget catastroph…
Objectobject-detectionObject DetectionRegion ProposalHSI Image Enhancement Classification Based on Knowledge Distillation: A Study on Forgetting
In incremental classification tasks for hyperspectral images, catastrophic forgetting is an unavoidable challenge. While memory recall methods can mitigate this issue, they heavily rely on samples from old categories. Th…
Knowledge DistillationImage ClassificationImage Enhancement