Knowledge Distillation: A Survey

2020-06-09 · Jianping Gou, Baosheng Yu, Stephen John Maybank, DaCheng Tao

In recent years, deep neural networks have been successful in both industry and academia, especially for computer vision tasks. The great success of deep learning is mainly due to its scalability to encode large-scale data and to maneuver billions of model parameters. However, it is a challenge to deploy these cumbersome deep models on devices with limited resources, e.g., mobile phones and embedded devices, not only because of the high computational complexity but also the large storage requirements. To this end, a variety of model compression and acceleration techniques have been developed. As a representative type of model compression and acceleration, knowledge distillation effectively learns a small student model from a large teacher model. It has received rapid increasing attention from the community. This paper provides a comprehensive survey of knowledge distillation from the perspectives of knowledge categories, training schemes, teacher-student architecture, distillation algorithms, performance comparison and applications. Furthermore, challenges in knowledge distillation are briefly reviewed and comments on future research are discussed and forwarded.

📄 PDF Abstract BibTeX arXiv:2006.05525

Code (0)

등록된 구현이 없습니다.

Tasks

Knowledge Distillation Model Compression Neural Architecture Search QuantizationSurvey

Methods 이 논문이 사용한 방법론

Knowledge Distillation A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions.…

A Comprehensive Survey on Knowledge Distillation

2025-03-15 · Amir M. Mansourian, Rozhan Ahmadi, Masoud Ghafouri, Amir Mohammad Babaei 외

Deep Neural Networks (DNNs) have achieved notable performance in the fields of computer vision and natural language processing with various applications in both academia and industry. However, with recent advancements in…

Knowledge Distillation Survey

A Selective Survey on Versatile Knowledge Distillation Paradigm for Neural Network Models

2020-11-30 · Jeong-Hoe Ku, Jihun Oh, YoungYoon Lee, Gaurav Pooniwala 외

This paper aims to provide a selective survey about knowledge distillation(KD) framework for researchers and practitioners to take advantage of it for developing new optimized models in the deep neural network field. To …

Knowledge Distillation Model Compression Self-Supervised Learning Survey

Knowledge Distillation in Deep Learning and its Applications

2020-07-17 · Abdolmaged Alkhulaifi, Fahad Alsahli, Irfan Ahmad

Deep learning based models are relatively large, and it is hard to deploy such models on resource-limited devices such as mobile phones and embedded devices. One possible solution is knowledge distillation whereby a smal…

Deep Learning Knowledge Distillation Survey

Teacher-Student Architecture for Knowledge Distillation: A Survey

2023-08-08 · Chengming Hu, Xuan Li, Dan Liu, Haolun Wu 외

Although Deep neural networks (DNNs) have shown a strong capacity to solve large-scale problems in many areas, such DNNs are hard to be deployed in real-world systems due to their voluminous parameters. To tackle this is…

Knowledge Distillation regression Survey

A Survey on Symbolic Knowledge Distillation of Large Language Models

2024-07-12 · Kamal Acharya, Alvaro Velasquez, Houbing Herbert Song

This survey paper delves into the emerging and critical area of symbolic knowledge distillation in Large Language Models (LLMs). As LLMs like Generative Pre-trained Transformer-3 (GPT-3) and Bidirectional Encoder Represe…