MLKD-BERT: Multi-level Knowledge Distillation for Pre-trained Language Models
Knowledge distillation is an effective technique for pre-trained language model compression. Although existing knowledge distillation methods perform well for the most typical model BERT, they could be further improved in two aspects: the relation-level knowledge could be further explored to improve model performance; and the setting of student attention head number could be more flexible to decrease inference time. Therefore, we are motivated to propose a novel knowledge distillation method MLKD-BERT to distill multi-level knowledge in teacher-student framework. Extensive experiments on GLUE benchmark and extractive question answering tasks demonstrate that our method outperforms state-of-the-art knowledge distillation methods on BERT. In addition, MLKD-BERT can flexibly set student attention head number, allowing for substantial inference time decrease with little performance drop.
Code (0)
등록된 구현이 없습니다.
Tasks
Extractive Question-AnsweringKnowledge DistillationLanguage ModelingLanguage ModellingModel CompressionQuestion AnsweringMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Multi-level Knowledge Distillation via Knowledge Alignment and Correlation
Knowledge distillation (KD) has become an important technique for model compression and knowledge transfer. In this work, we first perform a comprehensive analysis of the knowledge transferred by different KD methods. We…
Contrastive LearningKnowledge DistillationModel CompressionTransfer LearningMutual-Learning Knowledge Distillation for Nighttime UAV Tracking
Nighttime unmanned aerial vehicle (UAV) tracking has been facilitated with indispensable plug-and-play low-light enhancers. However, the introduction of low-light enhancers increases the extra computational burden for th…
Knowledge DistillationMulti-Level Knowledge Distillation and Dynamic Self-Supervised Learning for Continual Learning
Class-incremental with repetition (CIR), where previously trained classes repeatedly introduced in future tasks, is a more realistic scenario than the traditional class incremental setup, which assumes that each task con…
Self-Supervised LearningKnowledge DistillationContinual LearningLarge Language Model Teaches Visual Students: Cross-Modality Transfer of Fine-Grained Conceptual Knowledge
Large Language Models (LLMs) possess broad conceptual knowledge acquired through large-scale text pretraining, yet their potential to supervise models in other modalities remains underexplored. In this work, we propose L…
Knowledge DistillationSKDBERT: Compressing BERT via Stochastic Knowledge Distillation
In this paper, we propose Stochastic Knowledge Distillation (SKD) to obtain compact BERT-style language model dubbed SKDBERT. In each iteration, SKD samples a teacher model from a pre-defined teacher ensemble, which cons…
Knowledge DistillationLanguage ModelingLanguage Modelling