Tree-Like Decision Distillation
Knowledge distillation pursues a diminutive yet well-behaved student network by harnessing the knowledge learned by a cumbersome teacher model. Prior methods achieve this by making the student imitate shallow behaviors, such as soft targets, features, or attention, of the teacher. In this paper, we argue that what really matters for distillation is the intrinsic problem-solving process captured by the teacher. By dissecting the decision process in a layer-wise manner, we found that the decision-making procedure in the teacher model is conducted in a coarse-to-fine manner, where coarse-grained discrimination (e.g., animal vs vehicle) is attained in early layers, and fine-grained discrimination (e.g., dog vs cat, car vs truck) in latter layers. Motivated by this observation, we propose a new distillation method, dubbed as Tree-like Decision Distillation (TDD), to endow the student with the same problem-solving mechanism as that of the teacher. Extensive experiments demonstrated that TDD yields competitive performance compared to state of the arts. More importantly, it enjoys better interpretability due to its interpretable decision distillation instead of dark knowledge distillation.
Code (0)
등록된 구현이 없습니다.
Tasks
Decision MakingKnowledge DistillationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
A Rectification-Based Approach for Distilling Boosted Trees into Decision Trees
We present a new approach for distilling boosted trees into decision trees, in the objective of generating an ML model offering an acceptable compromise in terms of predictive performance and interpretability. We explain…
Rectified Decision Trees: Towards Interpretability, Compression and Empirical Soundness
How to obtain a model with good interpretability and performance has always been an important research topic. In this paper, we propose rectified decision trees (ReDT), a knowledge distillation based decision trees recti…
Knowledge DistillationMSVIPER: Improved Policy Distillation for Reinforcement-Learning-Based Robot Navigation
We present Multiple Scenario Verifiable Reinforcement Learning via Policy Extraction (MSVIPER), a new method for policy distillation to decision trees for improved robot navigation. MSVIPER learns an "expert" policy usin…
Imitation Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)+1Interpretable reinforcement learning for heat pump control through asymmetric differentiable decision trees
In recent years, deep reinforcement learning (DRL) algorithms have gained traction in home energy management systems. However, their adoption by energy management companies remains limited due to the black-box nature of …
Decision MakingDeep Reinforcement Learningenergy managementManagement+2Knowledge Distillation of Convolutional Neural Networks through Feature Map Transformation using Decision Trees
The interpretation of reasoning by Deep Neural Networks (DNN) is still challenging due to their perceived black-box nature. Therefore, deploying DNNs in several real-world tasks is restricted by the lack of transparency …
Knowledge Distillation