Improving Knowledge Distillation via Category Structure
Most previous knowledge distillation frameworks train the student to mimic the teacher's output of each sample or transfer cross-sample relations from the teacher to the student. Nevertheless, they neglect the structured relations at a category level. In this paper, a novel Category Structure is proposed to transfer category-level structured relations for knowledge distillation. It models two structured relations, including the intra-category structure and the inter-category structure, which are intrinsic natures in relations between samples. Intra-category structure penalizes the structured relations in samples from the same category and inter-category structure focuses on cross-category relations at a category level. Transferring category structure from the teacher to the student supplements category-level structured relations for training a better student. Extensive experiments show that our method groups samples from the same category tighter in the embedding space and the superiority of our method in comparison with closely related works are validated in different datasets and models.
Code (1)
Tasks
Knowledge DistillationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
HSI Image Enhancement Classification Based on Knowledge Distillation: A Study on Forgetting
In incremental classification tasks for hyperspectral images, catastrophic forgetting is an unavoidable challenge. While memory recall methods can mitigate this issue, they heavily rely on samples from old categories. Th…
Knowledge DistillationImage ClassificationImage EnhancementCategory Adaptation Meets Projected Distillation in Generalized Continual Category Discovery
Generalized Continual Category Discovery (GCCD) tackles learning from sequentially arriving, partially labeled datasets while uncovering new categories. Traditional methods depend on feature distillation to prevent forge…
class-incremental learningClass Incremental LearningContinual LearningIncremental Learning+2CaKDP: Category-aware Knowledge Distillation and Pruning Framework for Lightweight 3D Object Detection
Knowledge distillation (KD) possesses immense potential to accelerate the deep neural networks (DNNs) for LiDAR-based 3D detection. However in most of prevailing approaches the suboptimal teacher models and insuffici…
3D Object DetectionKnowledge Distillationobject-detectionObject DetectionWasserstein Distance Rivals Kullback-Leibler Divergence for Knowledge Distillation
Since pioneering work of Hinton et al., knowledge distillation based on Kullback-Leibler Divergence (KL-Div) has been predominant, and recently its variants have achieved compelling performance. However, KL-Div only comp…
image-classificationImage ClassificationKnowledge Distillationobject-detection+1Preview-based Category Contrastive Learning for Knowledge Distillation
Knowledge distillation is a mainstream algorithm in model compression by transferring knowledge from the larger model (teacher) to the smaller model (student) to improve the performance of student. Despite many efforts, …
Contrastive LearningKnowledge DistillationModel Compression