A Note on Knowledge Distillation Loss Function for Object Classification
This research note provides a quick introduction to the knowledge distillation loss function used in object classification. In particular, we discuss its connection to a previously proposed logits matching loss function. We further treat knowledge distillation as a specific form of output regularization and demonstrate its connection to label smoothing and entropy-based regularization.
Code (0)
등록된 구현이 없습니다.
Tasks
Knowledge DistillationModel CompressionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
An Empirical Study of Uniform-Architecture Knowledge Distillation in Document Ranking
Although BERT-based ranking models have been commonly used in commercial search engines, they are usually time-consuming for online ranking tasks. Knowledge distillation, which aims at learning a smaller model with compa…
Document RankingKnowledge DistillationAdaKD: Dynamic Knowledge Distillation of ASR models using Adaptive Loss Weighting
Knowledge distillation, a widely used model compression technique, works on the basis of transferring knowledge from a cumbersome teacher model to a lightweight student model. The technique involves jointly optimizing th…
Knowledge DistillationModel CompressionObject detection at 200 Frames Per Second
In this paper, we propose an efficient and fast object detector which can process hundreds of frames per second. To achieve this goal we investigate three main aspects of the object detection framework: network architect…
Objectobject-detectionObject DetectionDo Not Blindly Imitate the Teacher: Using Perturbed Loss for Knowledge Distillation
Knowledge distillation is a popular technique to transfer knowledge from large teacher models to a small student model. Typically, the student learns to imitate the teacher by minimizing the KL divergence of its output d…
Knowledge DistillationFeature Adversarial Distillation for Point Cloud Classification
Due to the point cloud's irregular and unordered geometry structure, conventional knowledge distillation technology lost a lot of information when directly used on point cloud tasks. In this paper, we propose Feature Adv…
ClassificationFADKnowledge DistillationModel Compression+2