Unified and Effective Ensemble Knowledge Distillation
Ensemble knowledge distillation can extract knowledge from multiple teacher models and encode it into a single student model. Many existing methods learn and distill the student model on labeled data only. However, the teacher models are usually learned on the same labeled data, and their predictions have high correlations with groudtruth labels. Thus, they cannot provide sufficient knowledge complementary to task labels for student teaching. Distilling on unseen unlabeled data has the potential to enhance the knowledge transfer from the teachers to the student. In this paper, we propose a unified and effective ensemble knowledge distillation method that distills a single student model from an ensemble of teacher models on both labeled and unlabeled data. Since different teachers may have diverse prediction correctness on the same sample, on labeled data we weight the predictions of different teachers according to their correctness. In addition, we weight the distillation loss based on the overall prediction correctness of the teacher ensemble to distill high-quality knowledge. On unlabeled data, there is no groundtruth to evaluate prediction correctness. Fortunately, the disagreement among teachers is an indication of sample hardness, and thereby we weight the distillation loss based on teachers' disagreement to emphasize knowledge distillation on important samples. Extensive experiments on four datasets show the effectiveness of our proposed ensemble distillation method.
Code (0)
등록된 구현이 없습니다.
Tasks
Knowledge DistillationTransfer LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
PILE: Pairwise Iterative Logits Ensemble for Multi-Teacher Labeled Distillation
Pre-trained language models have become a crucial part of ranking systems and achieved very impressive effects recently. To maintain high performance while keeping efficient computations, knowledge distillation is widely…
Knowledge DistillationMimic and Conquer: Heterogeneous Tree Structure Distillation for Syntactic NLP
Syntax has been shown useful for various NLP tasks, while existing work mostly encodes singleton syntactic tree using one hierarchical neural network. In this paper, we investigate a simple and effective method, Knowledg…
Knowledge DistillationPeer Collaborative Learning for Online Knowledge Distillation
Traditional knowledge distillation uses a two-stage training strategy to transfer knowledge from a high-capacity teacher model to a compact student model, which relies heavily on the pre-trained teacher. Recent online kn…
Knowledge DistillationDistilling Inductive Bias: Knowledge Distillation Beyond Model Compression
With the rapid development of computer vision, Vision Transformers (ViTs) offer the tantalizing prospect of unified information processing across visual and textual domains. But due to the lack of inherent inductive bias…
Inductive BiasKnowledge DistillationmodelModel CompressionTowards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning
We formally study how ensemble of deep learning models can improve test accuracy, and how the superior performance of ensemble can be distilled into a single model using knowledge distillation. We consider the challengin…
Deep LearningKnowledge DistillationLearning Theory