Information Theoretic Representation Distillation

2021-12-01 · Roy Miles, Adrian Lopez Rodriguez, Krystian Mikolajczyk

Despite the empirical success of knowledge distillation, current state-of-the-art methods are computationally expensive to train, which makes them difficult to adopt in practice. To address this problem, we introduce two distinct complementary losses inspired by a cheap entropy-like estimator. These losses aim to maximise the correlation and mutual information between the student and teacher representations. Our method incurs significantly less training overheads than other approaches and achieves competitive performance to the state-of-the-art on the knowledge distillation and cross-model transfer tasks. We further demonstrate the effectiveness of our method on a binary distillation task, whereby it leads to a new state-of-the-art for binary quantisation and approaches the performance of a full precision model. Code: www.github.com/roymiles/ITRD

📄 PDF Abstract BibTeX arXiv:2112.00459

Code (1)

roymiles/ITRD 공식 구현 pytorch

Tasks

Classification with Binary Weight NetworkKnowledge Distillation Question Answering

Methods 이 논문이 사용한 방법론

Knowledge Distillation A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions.…

Quantifying Knowledge Distillation Using Partial Information Decomposition

2024-11-12 · Pasan Dissanayake, Faisal Hamman, Barproda Halder, Ilia Sucholutsky 외

Knowledge distillation deploys complex machine learning models in resource-constrained environments by training a smaller student model to emulate internal representations of a complex teacher model. However, the teacher…

Knowledge Distillation Transfer Learning

Farewell to Mutual Information: Variational Distillation for Cross-Modal Person Re-Identification

2021-04-07 · CVPR 2021 1 · Xudong Tian, Zhizhong Zhang, Shaohui Lin, Yanyun Qu 외

The Information Bottleneck (IB) provides an information theoretic principle for representation learning, by retaining all information relevant for predicting label while minimizing the redundancy. Though IB principle has…

Cross-Modality Person Re-identification Cross-Modal Person Re-Identification Cross-Modal Person Re-Identification MULTI-VIEW LEARNING+2

Unifying distillation and privileged information

2015-11-11 · David Lopez-Paz, Léon Bottou, Bernhard Schölkopf, Vladimir Vapnik

Distillation (Hinton et al., 2015) and privileged information (Vapnik & Izmailov, 2015) are two techniques that enable machines to learn from other machines. This paper unifies these two techniques into generalized disti…

Demystifying Low-Rank Knowledge Distillation in Large Language Models: Convergence, Generalization, and Information-Theoretic Guarantees

2026-03-22 · Alberlucia Rafael Soarez, Daniel Kim, Mariana Costa, Alejandro Torre arxiv

Knowledge distillation has emerged as a powerful technique for compressing large language models (LLMs) into efficient, deployable architectures while preserving their advanced capabilities. Recent advances in low-rank k…

Knowledge Distillation Model Compression

Information-Theoretic Criteria for Knowledge Distillation in Multimodal Learning

2025-10-15 · Rongrong Xie, Yizhou Xu, Guido Sanguinetti arxiv

The rapid increase in multimodal data availability has sparked significant interest in cross-modal knowledge distillation (KD) techniques, where richer "teacher" modalities transfer information to weaker "student" modali…

Knowledge Distillation