What is Lost in Knowledge Distillation?
Deep neural networks (DNNs) have improved NLP tasks significantly, but training and maintaining such networks could be costly. Model compression techniques, such as, knowledge distillation (KD), have been proposed to address the issue; however, the compression process could be lossy. Motivated by this, our work investigates how a distilled student model differs from its teacher, if the distillation process causes any information losses, and if the loss follows a specific pattern. Our experiments aim to shed light on the type of tasks might be less or more sensitive to KD by reporting data points on the contribution of different factors, such as the number of layers or attention heads. Results such as ours could be utilized when determining effective and efficient configurations to achieve optimal information transfers between larger (teacher) and smaller (student) models.
Code (0)
등록된 구현이 없습니다.
Tasks
Knowledge DistillationModel CompressionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Knowledge Distillation Must Account for What It Loses
This position paper argues that knowledge distillation must account for what it loses: student models should be judged not only by retained task scores, but by whether they preserve the teacher capabilities that make tho…
Knowledge DistillationLost in Distillation: A Case Study in Toxicity Modeling
In an era of increasingly large pre-trained language models, knowledge distillation is a powerful tool for transferring information from a large model to a smaller one. In particular, distillation is of tremendous benefi…
Knowledge DistillationFeature Adversarial Distillation for Point Cloud Classification
Due to the point cloud's irregular and unordered geometry structure, conventional knowledge distillation technology lost a lot of information when directly used on point cloud tasks. In this paper, we propose Feature Adv…
ClassificationFADKnowledge DistillationModel Compression+2What Knowledge Gets Distilled in Knowledge Distillation?
Knowledge distillation aims to transfer useful information from a teacher network to a student network, with the primary goal of improving the student's performance for the task at hand. Over the years, there has a been …
Knowledge DistillationContinual Distillation of Teachers from Different Domains
Deep learning models continue to scale, with some requiring more storage than many large-scale datasets. Thus, we introduce a new paradigm: Continual Distillation (CD), where a student learns sequentially from a stream o…
Domain Generalization