Subclass Distillation
After a large "teacher" neural network has been trained on labeled data, the probabilities that the teacher assigns to incorrect classes reveal a lot of information about the way in which the teacher generalizes. By training a small "student" model to match these probabilities, it is possible to transfer most of the generalization ability of the teacher to the student, often producing a much better small model than directly training the student on the training data. The transfer works best when there are many possible classes because more is then revealed about the function learned by the teacher, but in cases where there are only a few possible classes we show that we can improve the transfer by forcing the teacher to divide each class into many subclasses that it invents during the supervised training. The student is then trained to match the subclass probabilities. For datasets where there are known, natural subclasses we demonstrate that the teacher learns similar subclasses and these improve distillation. For clickthrough datasets where the subclasses are unknown we demonstrate that subclass distillation allows the student to learn faster and better.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Subclass Knowledge Distillation with Known Subclass Labels
This work introduces a novel knowledge distillation framework for classification tasks where information on existing subclasses is available and taken into consideration. In classification tasks with a small number of cl…
Binary ClassificationKnowledge DistillationOn the Efficiency of Subclass Knowledge Distillation in Classification Tasks
This work introduces a novel knowledge distillation framework for classification tasks where information on existing subclasses is available and taken into consideration. In classification tasks with a small number of cl…
Binary ClassificationClassificationKnowledge DistillationMulti-Teacher Knowledge Distillation for Incremental Implicitly-Refined Classification
Incremental learning methods can learn new classes continually by distilling knowledge from the last model (as a teacher model) to the current model (as a student model) in the sequentially learning process. However, the…
ClassificationIncremental LearningKnowledge DistillationSelf-Distillation is Optimal Among Spectral Shrinkage Estimators in Spiked Covariance Models
Self-distillation has emerged as a promising technique for improving model performance in modern machine learning systems. We develop the statistical foundations of self-distillation in spiked covariance models, by intro…
No Subclass Left Behind: Fine-Grained Robustness in Coarse-Grained Classification Problems
In real-world classification tasks, each class often comprises multiple finer-grained "subclasses." As the subclass labels are frequently unavailable, models trained using only the coarser-grained class labels often exhi…
ClusteringGeneral Classificationimage-classificationImage Classification