Spirit Distillation: A Model Compression Method with Multi-domain Knowledge Transfer

2021-04-29 · Zhiyuan Wu, Yu Jiang, Minghao Zhao, Chupeng Cui, Zongmin Yang, Xinhui Xue, Hong Qi

Recent applications pose requirements of both cross-domain knowledge transfer and model compression to machine learning models due to insufficient training data and limited computational resources. In this paper, we propose a new knowledge distillation model, named Spirit Distillation (SD), which is a model compression method with multi-domain knowledge transfer. The compact student network mimics out a representation equivalent to the front part of the teacher network, through which the general knowledge can be transferred from the source domain (teacher) to the target domain (student). To further improve the robustness of the student, we extend SD to Enhanced Spirit Distillation (ESD) in exploiting a more comprehensive knowledge by introducing the proximity domain which is similar to the target domain for feature extraction. Results demonstrate that our method can boost mIOU and high-precision accuracy by 1.4% and 8.2% respectively with 78.2% segmentation variance, and can gain a precise compact network with only 41.8% FLOPs.

📄 PDF Abstract BibTeX arXiv:2104.14696

Code (0)

등록된 구현이 없습니다.

Tasks

General Knowledge Knowledge Distillation Model Compression Transfer Learning

Methods 이 논문이 사용한 방법론

Knowledge Distillation A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions.…

Spirit Distillation: Precise Real-time Semantic Segmentation of Road Scenes with Insufficient Data

2021-03-25 · Zhiyuan Wu, Yu Jiang, Chupeng Cui, Zongmin Yang 외

Semantic segmentation of road scenes is one of the key technologies for realizing autonomous driving scene perception, and the effectiveness of deep Convolutional Neural Networks(CNNs) for this task has been demonstrated…

Autonomous Driving Few-Shot Learning Knowledge Distillation Real-Time Semantic Segmentation+3

Meta-KD: A Meta Knowledge Distillation Framework for Language Model Compression across Domains

2020-12-02 · ACL 2021 5 · Haojie Pan, Chengyu Wang, Minghui Qiu, Yichang Zhang 외

Pre-trained language models have been applied to various NLP tasks with considerable performance gains. However, the large model sizes, together with the long inference time, limit the deployment of such models in real-t…

Knowledge Distillation Language Modeling Language Modelling Meta-Learning+2

Scaling Laws for Task-Specific LLM Distillation

2026-06-23 · Lavinia Ghita, Dhruv Desai, Ioana Boier arxiv

Large Language Models (LLMs) achieve strong performance across a growing range of domains, yet their scale poses deployment challenges in applications where latency and cost constraints are critical. This paper derives e…

General Knowledge

Sparse Knowledge Distillation: A Mathematical Framework for Probability-Domain Temperature Scaling and Multi-Stage Compression

2026-01-06 · Aaron R. Flouro, Shawn P. Chadwick arxiv

We develop a unified theoretical framework for sparse knowledge distillation based on probability-domain softening operators. While the equivalence $p^{1/T} \propto \mathrm{softmax}(z/T)$ is well known, our contribution …

Knowledge Distillation Model Compression

Weight Averaging Improves Knowledge Distillation under Domain Shift

2023-09-20 · Valeriy Berezovskiy, Nikita Morozov

Knowledge distillation (KD) is a powerful model compression technique broadly used in practical deep learning applications. It is focused on training a small student network to mimic a larger teacher network. While it is…

Domain Generalization Knowledge Distillation