CrossKD: Cross-Head Knowledge Distillation for Object Detection
Knowledge Distillation (KD) has been validated as an effective model compression technique for learning compact object detectors. Existing state-of-the-art KD methods for object detection are mostly based on feature imitation. In this paper, we present a general and effective prediction mimicking distillation scheme, called CrossKD, which delivers the intermediate features of the student's detection head to the teacher's detection head. The resulting cross-head predictions are then forced to mimic the teacher's predictions. This manner relieves the student's head from receiving contradictory supervision signals from the annotations and the teacher's predictions, greatly improving the student's detection performance. Moreover, as mimicking the teacher's predictions is the target of KD, CrossKD offers more task-oriented information in contrast with feature imitation. On MS COCO, with only prediction mimicking losses applied, our CrossKD boosts the average precision of GFL ResNet-50 with 1x training schedule from 40.2 to 43.7, outperforming all existing KD methods. In addition, our method also works well when distilling detectors with heterogeneous backbones. Code is available at https://github.com/jbwang1997/CrossKD.
Code (1)
Tasks
Dense Object DetectionKnowledge DistillationModel CompressionObjectobject-detectionObject DetectionSimilar Papers 제목 키워드 기반
Cross-Architecture Distillation Made Simple with Redundancy Suppression
We describe a simple method for cross-architecture knowledge distillation, where the knowledge transfer is cast into a redundant information suppression formulation. Existing methods introduce sophisticated modules, arch…
Knowledge DistillationNoOVD: Novel Category Discovery and Embedding for Open-Vocabulary Object Detection
Despite the remarkable progress in open-vocabulary object detection (OVD), a significant gap remains between the training and testing phases. During training, the RPN and RoI heads often misclassify unlabeled novel-categ…
Knowledge DistillationObject DetectionUniKD: Universal Knowledge Distillation for Mimicking Homogeneous or Heterogeneous Object Detectors
Knowledge distillation (KD) has become a standard method to boost the performance of lightweight object detectors. Most previous works are feature-based, where students mimic the features of homogeneous teacher detec…
Knowledge DistillationOvercoming Catastrophic Forgetting in Incremental Object Detection via Elastic Response Distillation
Traditional object detectors are ill-equipped for incremental learning. However, fine-tuning directly on a well-trained detection model with only new data will lead to catastrophic forgetting. Knowledge distillation is a…
Class-Incremental Object DetectionIncremental LearningKnowledge Distillationobject-detection+1FD-CanKD: Frequency-Decoupled Cross-Attention Distillation as a Refinement Prior for Compact Object Detectors
Compact object detectors are suitable for resource-constrained visual perception, but their limited representation capacity creates an accuracy gap relative to large models. Conventional detector distillation often relie…
Knowledge Distillation