On the Impact of Knowledge Distillation for Model Interpretability
Several recent studies have elucidated why knowledge distillation (KD) improves model performance. However, few have researched the other advantages of KD in addition to its improving model performance. In this study, we have attempted to show that KD enhances the interpretability as well as the accuracy of models. We measured the number of concept detectors identified in network dissection for a quantitative comparison of model interpretability. We attributed the improvement in interpretability to the class-similarity information transferred from the teacher to student models. First, we confirmed the transfer of class-similarity information from the teacher to student model via logit distillation. Then, we analyzed how class-similarity information affects model interpretability in terms of its presence or absence and degree of similarity information. We conducted various quantitative and qualitative experiments and examined the results on different datasets, different KD methods, and according to different measures of interpretability. Our research showed that KD models by large models could be used more reliably in various fields.
Code (0)
등록된 구현이 없습니다.
Tasks
Knowledge DistillationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Class Attention Transfer Based Knowledge Distillation
Previous knowledge distillation methods have shown their impressive performance on model compression tasks, however, it is hard to explain how the knowledge they transferred helps to improve the performance of the studen…
Knowledge DistillationModel CompressionKnowledge Distillation with Adapted Weight
Although large models have shown a strong capacity to solve large-scale problems in many areas including natural language and computer vision, their voluminous parameters are hard to deploy in a real-time system due to c…
4kFairnessKnowledge DistillationRectified Decision Trees: Towards Interpretability, Compression and Empirical Soundness
How to obtain a model with good interpretability and performance has always been an important research topic. In this paper, we propose rectified decision trees (ReDT), a knowledge distillation based decision trees recti…
Knowledge DistillationKGEx: Explaining Knowledge Graph Embeddings via Subgraph Sampling and Knowledge Distillation
Despite being the go-to choice for link prediction on knowledge graphs, research on interpretability of knowledge graph embeddings (KGE) has been relatively unexplored. We present KGEx, a novel post-hoc method that expla…
Knowledge DistillationKnowledge Graph EmbeddingsKnowledge GraphsLink PredictionA Functional Perspective on Knowledge Distillation in Neural Networks
Knowledge distillation is considered a compression mechanism when judged on the resulting student's accuracy and loss, yet its functional impact is poorly understood. We quantify the compression capacity of knowledge dis…
Knowledge Distillation