AD-KD: Attribution-Driven Knowledge Distillation for Language Model Compression
Knowledge distillation has attracted a great deal of interest recently to compress pre-trained language models. However, existing knowledge distillation methods suffer from two limitations. First, the student model simply imitates the teacher's behavior while ignoring the underlying reasoning. Second, these methods usually focus on the transfer of sophisticated model-specific knowledge but overlook data-specific knowledge. In this paper, we present a novel attribution-driven knowledge distillation approach, which explores the token-level rationale behind the teacher model based on Integrated Gradients (IG) and transfers attribution knowledge to the student model. To enhance the knowledge transfer of model reasoning and generalization, we further explore multi-view attribution distillation on all potential decisions of the teacher. Comprehensive experiments are conducted with BERT on the GLUE benchmark. The experimental results demonstrate the superior performance of our approach to several state-of-the-art methods.
Code (1)
Tasks
Knowledge DistillationLanguage ModelingLanguage ModellingModel CompressionTransfer LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
A Functional Perspective on Knowledge Distillation in Neural Networks
Knowledge distillation is considered a compression mechanism when judged on the resulting student's accuracy and loss, yet its functional impact is poorly understood. We quantify the compression capacity of knowledge dis…
Knowledge DistillationAutoregressive Knowledge Distillation through Imitation Learning
The performance of autoregressive models on natural language generation tasks has dramatically improved due to the adoption of deep, self-attentive architectures. However, these gains have come at the cost of hindering i…
Imitation LearningKnowledge DistillationMachine TranslationText Generation+1Contrastive Distillation on Intermediate Representations for Language Model Compression
Existing language model compression methods mostly use a simple L2 loss to distill knowledge in the intermediate representations of a large BERT model to a smaller one. Although widely used, this objective by design assu…
Knowledge DistillationLanguage ModelingLanguage ModellingModel CompressionLadaBERT: Lightweight Adaptation of BERT through Hybrid Model Compression
BERT is a cutting-edge language representation model pre-trained by a large corpus, which achieves superior performances on various natural language understanding tasks. However, a major blocking issue of applying BERT t…
BlockingKnowledge DistillationModel CompressionNatural Language UnderstandingEnergy-efficient Knowledge Distillation for Spiking Neural Networks
Spiking neural networks (SNNs) have been gaining interest as energy-efficient alternatives of conventional artificial neural networks (ANNs) due to their event-driven computation. Considering the future deployment of SNN…
Knowledge DistillationModel CompressionQuantization