D$^3$ETR: Decoder Distillation for Detection Transformer
While various knowledge distillation (KD) methods in CNN-based detectors show their effectiveness in improving small students, the baselines and recipes for DETR-based detectors are yet to be built. In this paper, we focus on the transformer decoder of DETR-based detectors and explore KD methods for them. The outputs of the transformer decoder lie in random order, which gives no direct correspondence between the predictions of the teacher and the student, thus posing a challenge for knowledge distillation. To this end, we propose MixMatcher to align the decoder outputs of DETR-based teachers and students, which mixes two teacher-student matching strategies, i.e., Adaptive Matching and Fixed Matching. Specifically, Adaptive Matching applies bipartite matching to adaptively match the outputs of the teacher and the student in each decoder layer, while Fixed Matching fixes the correspondence between the outputs of the teacher and the student with the same object queries, with the teacher's fixed object queries fed to the decoder of the student as an auxiliary group. Based on MixMatcher, we build \textbf{D}ecoder \textbf{D}istillation for \textbf{DE}tection \textbf{TR}ansformer (D$^3$ETR), which distills knowledge in decoder predictions and attention maps from the teachers to students. D$^3$ETR shows superior performance on various DETR-based detectors with different backbones. For example, D$^3$ETR improves Conditional DETR-R50-C5 by $\textbf{7.8}/\textbf{2.4}$ mAP under $12/50$ epochs training settings with Conditional DETR-R101-C5 as the teacher.
Code (0)
등록된 구현이 없습니다.
Tasks
DecoderKnowledge DistillationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Distillation Using Oracle Queries for Transformer-Based Human-Object Interaction Detection
Transformer-based methods have achieved great success in the field of human-object interaction (HOI) detection. However, these models tend to adopt semantically ambiguous queries, which lowers the transformer's repre…
Data AugmentationDecoderHuman-Object Interaction DetectionKnowledge Distillation+1CLoCKDistill: Consistent Location-and-Context-aware Knowledge Distillation for DETRs
Object detection has advanced significantly with Detection Transformers (DETRs). However, these models are computationally demanding, posing challenges for deployment in resource-constrained environments (e.g., self-driv…
DenoisingKnowledge Distillationobject-detectionObject Detection+1Li3DeTr: A LiDAR based 3D Detection Transformer
Inspired by recent advances in vision transformers for object detection, we propose Li3DeTr, an end-to-end LiDAR based 3D Detection Transformer for autonomous driving, that inputs LiDAR point clouds and regresses 3D boun…
Autonomous DrivingDecoderKnowledge DistillationObject+2Accelerate 3D Object Detection Models via Zero-Shot Attention Key Pruning
Query-based methods with dense features have demonstrated remarkable success in 3D object detection tasks. However, the computational demands of these models, particularly with large image sizes and multiple transformer …
3D Object Detectionobject-detectionObject DetectionJoint Multiple Intent Detection and Slot Filling via Self-distillation
Intent detection and slot filling are two main tasks in natural language understanding (NLU) for identifying users' needs from their utterances. These two tasks are highly related and often trained jointly. However, most…
DecoderIntent DetectionMultiple Instance LearningNatural Language Understanding+2