MKD: a Multi-Task Knowledge Distillation Approach for Pretrained Language Models
Pretrained language models have led to significant performance gains in many NLP tasks. However, the intensive computing resources to train such models remain an issue. Knowledge distillation alleviates this problem by learning a light-weight student model. So far the distillation approaches are all task-specific. In this paper, we explore knowledge distillation under the multi-task learning setting. The student is jointly distilled across different tasks. It acquires more general representation capacity through multi-tasking distillation and can be further fine-tuned to improve the model in the target domain. Unlike other BERT distillation methods which specifically designed for Transformer-based architectures, we provide a general learning framework. Our approach is model agnostic and can be easily applied on different future teacher model architectures. We evaluate our approach on a Transformer-based and LSTM based student model. Compared to a strong, similarly LSTM-based approach, we achieve better quality under the same computational constraints. Compared to the present state of the art, we reach comparable results with much faster inference speed.
Code (0)
등록된 구현이 없습니다.
Tasks
Knowledge DistillationMulti-Task LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
ERNIE 3.0 Tiny: Frustratingly Simple Method to Improve Task-Agnostic Distillation Generalization
Task-agnostic knowledge distillation attempts to address the problem of deploying large pretrained language model in resource-constrained scenarios by compressing a large pretrained model called teacher into a smaller on…
Knowledge DistillationLanguage ModellingMulti-Task LearningLimitations of Knowledge Distillation for Zero-shot Transfer Learning
Pretrained transformer-based encoders such as BERT have been demonstrated to achieve state-of-the-art performance on numerous NLP tasks. Despite their success, BERT style encoders are large in size and have high latency …
CPUCross-Lingual TransferKnowledge DistillationTransfer Learning+1Natural Language Generation for Effective Knowledge Distillation
Knowledge distillation can effectively transfer knowledge from BERT, a deep language representation model, to traditional, shallow word embedding-based neural networks, helping them approach or exceed the quality of othe…
Knowledge DistillationLinguistic AcceptabilitySentenceSentence Similarity+4Domain Knowledge Transferring for Pre-trained Language Model via Calibrated Activation Boundary Distillation
Since the development and wide use of pretrained language models (PLMs), several approaches have been applied to boost their performance on downstream tasks in specific domains, such as biomedical or scientific domains. …
Knowledge DistillationLanguage ModelingLanguage ModellingSelf-Knowledge Distillation in Natural Language Processing
Since deep learning became a key player in natural language processing (NLP), many deep learning models have been showing remarkable performances in a variety of NLP tasks, and in some cases, they are even outperforming …
Deep LearningKnowledge DistillationLanguage ModelingLanguage Modelling+3