Explicit and Implicit Knowledge Distillation via Unlabeled Data
Data-free knowledge distillation is a challenging model lightweight task for scenarios in which the original dataset is not available. Previous methods require a lot of extra computational costs to update one or more generators and their naive imitate-learning lead to lower distillation efficiency. Based on these observations, we first propose an efficient unlabeled sample selection method to replace high computational generators and focus on improving the training efficiency of the selected samples. Then, a class-dropping mechanism is designed to suppress the label noise caused by the data domain shifts. Finally, we propose a distillation method that incorporates explicit features and implicit structured relations to improve the effect of distillation. Experimental results show that our method can quickly converge and obtain higher accuracy than other state-of-the-art methods.
Code (0)
등록된 구현이 없습니다.
Tasks
Data-free Knowledge DistillationKnowledge DistillationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Adaptive Explicit Knowledge Transfer for Knowledge Distillation
Logit-based knowledge distillation (KD) for classification is cost-efficient compared to feature-based KD but often subject to inferior performance. Recently, it was shown that the performance of logit-based KD can be im…
Knowledge DistillationTransfer LearningLow Resource Causal Event Detection from Biomedical Literature
Recognizing causal precedence relations among the chemical interactions in biomedical literature is crucial to understanding the underlying biological mechanisms. However, detecting such causal relation can be hard becau…
Event DetectionKnowledge DistillationRelationStereo-Knowledge Distillation from dpMV to Dual Pixels for Light Field Video Reconstruction
Dual pixels contain disparity cues arising from the defocus blur. This disparity information is useful for many vision tasks ranging from autonomous driving to 3D creative realism. However, directly estimating disparity …
Autonomous DrivingKnowledge DistillationVideo ReconstructionFeature Interaction Fusion Self-Distillation Network For CTR Prediction
Click-Through Rate (CTR) prediction plays a vital role in recommender systems, online advertising, and search engines. Most of the current approaches model feature interactions through stacked or parallel structures, wit…
Click-Through Rate PredictionKnowledge DistillationModel CompressionPrediction+2Dual-Stream Spectral Decoupling Distillation for Remote Sensing Object Detection
Knowledge distillation is an effective and hardware-friendly method, which plays a key role in lightweighting remote sensing object detection. However, existing distillation methods often encounter the issue of mixed fea…
Knowledge DistillationSmall Object Detection