Hard Gate Knowledge Distillation -- Leverage Calibration for Robust and Reliable Language Model
In knowledge distillation, a student model is trained with supervisions from both knowledge from a teacher and observations drawn from a training data distribution. Knowledge of a teacher is considered a subject that holds inter-class relations which send a meaningful supervision to a student; hence, much effort has been put to find such knowledge to be distilled. In this paper, we explore a question that has been given little attention: "when to distill such knowledge." The question is answered in our work with the concept of model calibration; we view a teacher model not only as a source of knowledge but also as a gauge to detect miscalibration of a student. This simple and yet novel view leads to a hard gate knowledge distillation scheme that switches between learning from a teacher model and training data. We verify the gating mechanism in the context of natural language generation at both the token-level and the sentence-level. Empirical comparisons with strong baselines show that hard gate knowledge distillation not only improves model generalization, but also significantly lowers model calibration error.
Code (0)
등록된 구현이 없습니다.
Tasks
Knowledge DistillationLanguage ModelingLanguage ModellingSentenceText GenerationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Teacher Network Calibration Improves Cross-Quality Knowledge Distillation
We investigate cross-quality knowledge distillation (CQKD), a knowledge distillation method where knowledge from a teacher network trained with full-resolution images is transferred to a student network that takes as inp…
image-classificationImage ClassificationKnowledge DistillationKDCTime: Knowledge Distillation with Calibration on InceptionTime for Time-series Classification
Time-series classification approaches based on deep neural networks are easy to be overfitting on UCR datasets, which is caused by the few-shot problem of those datasets. Therefore, in order to alleviate the overfitting …
Knowledge DistillationTime SeriesTime Series AnalysisTime Series ClassificationDeep Probabilistic Supervision for Image Classification
Supervised training of deep neural networks for classification typically relies on hard targets, which promote overconfidence and can limit calibration, generalization, and robustness. Self-distillation methods aim to mi…
Image ClassificationMulti-to-Single Knowledge Distillation for Point Cloud Semantic Segmentation
3D point cloud semantic segmentation is one of the fundamental tasks for environmental understanding. Although significant progress has been made in recent years, the performance of classes with few examples or few point…
Knowledge DistillationSemantic SegmentationPLaD: Preference-based Large Language Model Distillation with Pseudo-Preference Pairs
Large Language Models (LLMs) have exhibited impressive capabilities in various tasks, yet their vast parameter sizes restrict their applicability in resource-constrained settings. Knowledge distillation (KD) offers a via…
Knowledge DistillationLanguage ModelingLanguage ModellingLarge Language Model