Knowledge Distillation in Deep Learning and its Applications
Deep learning based models are relatively large, and it is hard to deploy such models on resource-limited devices such as mobile phones and embedded devices. One possible solution is knowledge distillation whereby a smaller model (student model) is trained by utilizing the information from a larger model (teacher model). In this paper, we present a survey of knowledge distillation techniques applied to deep learning models. To compare the performances of different techniques, we propose a new metric called distillation metric. Distillation metric compares different knowledge distillation algorithms based on sizes and accuracy scores. Based on the survey, some interesting conclusions are drawn and presented in this paper.
Code (0)
등록된 구현이 없습니다.
Tasks
Deep LearningKnowledge DistillationSurveyMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
A Selective Survey on Versatile Knowledge Distillation Paradigm for Neural Network Models
This paper aims to provide a selective survey about knowledge distillation(KD) framework for researchers and practitioners to take advantage of it for developing new optimized models in the deep neural network field. To …
Knowledge DistillationModel CompressionSelf-Supervised LearningSurveyGraph-based Knowledge Distillation: A survey and experimental evaluation
Graph, such as citation networks, social networks, and transportation networks, are prevalent in the real world. Graph Neural Networks (GNNs) have gained widespread attention for their robust expressiveness and exception…
Knowledge DistillationSelf-Knowledge DistillationSurveyAdaptive Distillation: Aggregating Knowledge from Multiple Paths for Efficient Distillation
Knowledge Distillation is becoming one of the primary trends among neural network compression algorithms to improve the generalization performance of a smaller student model with guidance from a larger teacher model. Thi…
Knowledge DistillationNeural Network Compressionobject-detectionObject Detection+1GKD: A General Knowledge Distillation Framework for Large-scale Pre-trained Language Model
Currently, the reduction in the parameter scale of large-scale pre-trained language models (PLMs) through knowledge distillation has greatly facilitated their widespread deployment on various devices. However, the deploy…
General KnowledgeKnowledge DistillationLanguage ModelingLanguage ModellingA Comprehensive Review of Knowledge Distillation in Computer Vision
Deep learning techniques have been demonstrated to surpass preceding cutting-edge machine learning techniques in recent years, with computer vision being one of the most prominent examples. However, deep learning models …
Deep LearningKnowledge Distillation