Knowledge Distillation on Graphs: A Survey

2023-02-01 · Yijun Tian, Shichao Pei, Xiangliang Zhang, Chuxu Zhang, Nitesh V. Chawla

Graph Neural Networks (GNNs) have attracted tremendous attention by demonstrating their capability to handle graph data. However, they are difficult to be deployed in resource-limited devices due to model sizes and scalability constraints imposed by the multi-hop data dependency. In addition, real-world graphs usually possess complex structural information and features. Therefore, to improve the applicability of GNNs and fully encode the complicated topological information, knowledge distillation on graphs (KDG) has been introduced to build a smaller yet effective model and exploit more knowledge from data, leading to model compression and performance improvement. Recently, KDG has achieved considerable progress with many studies proposed. In this survey, we systematically review these works. Specifically, we first introduce KDG challenges and bases, then categorize and summarize existing works of KDG by answering the following three questions: 1) what to distillate, 2) who to whom, and 3) how to distillate. Finally, we share our thoughts on future research directions.

📄 PDF Abstract BibTeX arXiv:2302.00219

Code (0)

등록된 구현이 없습니다.

Tasks

Knowledge Distillation Model CompressionSurvey

Methods 이 논문이 사용한 방법론

Knowledge Distillation A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions.…

A Comprehensive Survey on Knowledge Distillation

2025-03-15 · Amir M. Mansourian, Rozhan Ahmadi, Masoud Ghafouri, Amir Mohammad Babaei 외

Deep Neural Networks (DNNs) have achieved notable performance in the fields of computer vision and natural language processing with various applications in both academia and industry. However, with recent advancements in…

Knowledge Distillation Survey

A Selective Survey on Versatile Knowledge Distillation Paradigm for Neural Network Models

2020-11-30 · Jeong-Hoe Ku, Jihun Oh, YoungYoon Lee, Gaurav Pooniwala 외

This paper aims to provide a selective survey about knowledge distillation(KD) framework for researchers and practitioners to take advantage of it for developing new optimized models in the deep neural network field. To …

Knowledge Distillation Model Compression Self-Supervised Learning Survey

Knowledge Distillation in Deep Learning and its Applications

2020-07-17 · Abdolmaged Alkhulaifi, Fahad Alsahli, Irfan Ahmad

Deep learning based models are relatively large, and it is hard to deploy such models on resource-limited devices such as mobile phones and embedded devices. One possible solution is knowledge distillation whereby a smal…

Deep Learning Knowledge Distillation Survey

Teacher-Student Architecture for Knowledge Distillation: A Survey

2023-08-08 · Chengming Hu, Xuan Li, Dan Liu, Haolun Wu 외

Although Deep neural networks (DNNs) have shown a strong capacity to solve large-scale problems in many areas, such DNNs are hard to be deployed in real-world systems due to their voluminous parameters. To tackle this is…

Knowledge Distillation regression Survey

A Survey on Symbolic Knowledge Distillation of Large Language Models

2024-07-12 · Kamal Acharya, Alvaro Velasquez, Houbing Herbert Song

This survey paper delves into the emerging and critical area of symbolic knowledge distillation in Large Language Models (LLMs). As LLMs like Generative Pre-trained Transformer-3 (GPT-3) and Bidirectional Encoder Represe…