Knowledge Distillation via Weighted Ensemble of Teaching Assistants
Knowledge distillation in machine learning is the process of transferring knowledge from a large model called the teacher to a smaller model called the student. Knowledge distillation is one of the techniques to compress the large network (teacher) to a smaller network (student) that can be deployed in small devices such as mobile phones. When the network size gap between the teacher and student increases, the performance of the student network decreases. To solve this problem, an intermediate model is employed between the teacher model and the student model known as the teaching assistant model, which in turn bridges the gap between the teacher and the student. In this research, we have shown that using multiple teaching assistant models, the student model (the smaller model) can be further improved. We combined these multiple teaching assistant models using weighted ensemble learning where we have used a differential evaluation optimization algorithm to generate the weight values.
Code (0)
등록된 구현이 없습니다.
Tasks
Ensemble LearningKnowledge DistillationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Densely Guided Knowledge Distillation using Multiple Teacher Assistants
With the success of deep neural networks, knowledge distillation which guides the learning of a small student network from a large teacher network is being actively studied for model compression and transfer learning. Ho…
Knowledge DistillationModel CompressionTransfer LearningCascaded channel pruning using hierarchical self-distillation
In this paper, we propose an approach for filter-level pruning with hierarchical knowledge distillation based on the teacher, teaching-assistant, and student framework. Our method makes use of teaching assistants at inte…
Knowledge DistillationModel CompressionSelf-supervised Models are Good Teaching Assistants for Vision Transformers
Transformers have shown remarkable progress on computer vision tasks in the past year. Compared to their CNN counterparts, transformers usually need the help of distillation to achieve comparable results on middle or sma…
Image ClassificationKnowledge DistillationPrivate Deep Learning with Teacher Ensembles
Privacy-preserving deep learning is crucial for deploying deep neural network based solutions, especially when the model works on data that contains sensitive information. Most privacy-preserving methods lead to undesira…
Deep LearningEnsemble LearningKnowledge DistillationPrivacy Preserving+1Unified and Effective Ensemble Knowledge Distillation
Ensemble knowledge distillation can extract knowledge from multiple teacher models and encode it into a single student model. Many existing methods learn and distill the student model on labeled data only. However, the t…
Knowledge DistillationTransfer Learning