Prune Your Model Before Distill It
Knowledge distillation transfers the knowledge from a cumbersome teacher to a small student. Recent results suggest that the student-friendly teacher is more appropriate to distill since it provides more transferable knowledge. In this work, we propose the novel framework, "prune, then distill," that prunes the model first to make it more transferrable and then distill it to the student. We provide several exploratory examples where the pruned teacher teaches better than the original unpruned networks. We further show theoretically that the pruned teacher plays the role of regularizer in distillation, which reduces the generalization error. Based on this result, we propose a novel neural network compression scheme where the student network is formed based on the pruned teacher and then apply the "prune, then distill" strategy. The code is available at https://github.com/ososos888/prune-then-distill
Code (1)
Tasks
Knowledge DistillationmodelNeural Network CompressionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
EPSD: Early Pruning with Self-Distillation for Efficient Model Compression
Neural network compression techniques, such as knowledge distillation (KD) and network pruning, have received increasing attention. Recent work `Prune, then Distill' reveals that a pruned student-friendly teacher network…
Knowledge DistillationModel CompressionNetwork PruningNeural Network CompressionDistill the Best, Ignore the Rest: Improving Dataset Distillation with Loss-Value-Based Pruning
Dataset distillation has gained significant interest in recent years, yet existing approaches typically distill from the entire dataset, potentially including non-beneficial samples. We introduce a novel "Prune First, Di…
Dataset DistillationWhy Can You Lay Off Heads? Investigating How BERT Heads Transfer
The huge size of the widely used BERT family models has led to recent efforts about model distillation. The main goal of distillation is to create a task-agnostic pre-trained model that can be fine-tuned on downstream ta…
Transfer LearningAutomated Channel Pruning with Learned Importance
Neural network pruning allows for significant reduction of model size and latency. However, most of the current network pruning methods do not consider channel interdependencies and a lot of manual adjustments are requir…
DenoisingGPUImage DenoisingKnowledge Distillation+1Self-Distillation with Meta Learning for Knowledge Graph Completion
In this paper, we propose a selfdistillation framework with meta learning(MetaSD) for knowledge graph completion with dynamic pruning, which aims to learn compressed graph embeddings and tackle the longtail samples. Spec…
Knowledge Graph CompletionMeta-LearningTransfer Learning