Graph-based Knowledge Distillation by Multi-head Attention Network

2019-07-04 · Seunghyun Lee, Byung Cheol Song

Knowledge distillation (KD) is a technique to derive optimal performance from a small student network (SN) by distilling knowledge of a large teacher network (TN) and transferring the distilled knowledge to the small SN. Since a role of convolutional neural network (CNN) in KD is to embed a dataset so as to perform a given task well, it is very important to acquire knowledge that considers intra-data relations. Conventional KD methods have concentrated on distilling knowledge in data units. To our knowledge, any KD methods for distilling information in dataset units have not yet been proposed. Therefore, this paper proposes a novel method that enables distillation of dataset-based knowledge from the TN using an attention network. The knowledge of the embedding procedure of the TN is distilled to graph by multi-head attention (MHA), and multi-task learning is performed to give relational inductive bias to the SN. The MHA can provide clear information about the source dataset, which can greatly improves the performance of the SN. Experimental results show that the proposed method is 7.05% higher than the SN alone for CIFAR100, which is 2.46% higher than the state-of-the-art.

📄 PDF Abstract BibTeX arXiv:1907.02226

Code (2)

sseung0703/Knowledge_distillation_via_TF2.0 공식 구현 tf

sseung0703/KD_methods_with_TF tf

Tasks

Inductive BiasKnowledge Distillation Multi-Task Learning Transfer Learning

Methods 이 논문이 사용한 방법론

Attention 설명 없음

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Multi-Head Attention 설명 없음

Attention is all you need for boosting graph convolutional neural network

2024-03-10 · Yinwei Wu

Graph Convolutional Neural Networks (GCNs) possess strong capabilities for processing graph data in non-grid domains. They can capture the topological logical structure and node features in graphs and integrate them into…

All Knowledge Distillation Recommendation Systems

MLKD-BERT: Multi-level Knowledge Distillation for Pre-trained Language Models

2024-07-03 · Ying Zhang, Ziheng Yang, Shufan Ji

Knowledge distillation is an effective technique for pre-trained language model compression. Although existing knowledge distillation methods perform well for the most typical model BERT, they could be further improved i…

Extractive Question-Answering Knowledge Distillation Language Modeling Language Modelling+2

Optimizing Knowledge Distillation in Transformers: Enabling Multi-Head Attention without Alignment Barriers

2025-02-11 · Zhaodong Bing, Linze Li, Jiajun Liang

Knowledge distillation (KD) in transformers often faces challenges due to misalignment in the number of attention heads between teacher and student models. Existing methods either require identical head counts or introdu…

image-classification Image Classification Image Generation Knowledge Distillation+1

MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers

2020-12-31 · Findings (ACL) 2021 8 · Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong 외

We generalize deep self-attention distillation in MiniLM (Wang et al., 2020) by only using self-attention relation distillation for task-agnostic compression of pretrained Transformers. In particular, we define multi-hea…

Relation XLM-R

Efficient Mathematical Reasoning Models via Dynamic Pruning and Knowledge Distillation

2025-11-15 · Fengming Yu, Qingyu Meng, Haiwei Pan, Kejia Zhang arxiv

With the rapid development of deep learning, large language models have shown strong capabilities in complex reasoning tasks such as mathematical equation solving. However, their substantial computational and storage cos…

Knowledge Distillation Mathematical Reasoning