Bridging the Gap between Decision and Logits in Decision-based Knowledge Distillation for Pre-trained Language Models
Conventional knowledge distillation (KD) methods require access to the internal information of teachers, e.g., logits. However, such information may not always be accessible for large pre-trained language models (PLMs). In this work, we focus on decision-based KD for PLMs, where only teacher decisions (i.e., top-1 labels) are accessible. Considering the information gap between logits and decisions, we propose a novel method to estimate logits from the decision distributions. Specifically, decision distributions can be both derived as a function of logits theoretically and estimated with test-time data augmentation empirically. By combining the theoretical and empirical estimations of the decision distributions together, the estimation of logits can be successfully reduced to a simple root-finding problem. Extensive experiments show that our method significantly outperforms strong baselines on both natural language understanding and machine reading comprehension datasets.
Code (1)
Tasks
Data AugmentationKnowledge DistillationMachine Reading ComprehensionNatural Language UnderstandingReading ComprehensionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
FiGKD: Fine-Grained Knowledge Distillation via High-Frequency Detail Transfer
Knowledge distillation (KD) is a widely adopted technique for transferring knowledge from a high-capacity teacher model to a smaller student model by aligning their output distributions. However, existing methods often u…
Fine-Grained Visual RecognitionKnowledge DistillationTransfer LearningConformalized Decision Risk Assessment
High-stakes decisions in domains such as healthcare, energy, and public policy are often made by human experts using domain knowledge and heuristics, yet are increasingly supported by predictive and optimization-based to…
Conformal PredictionAvoid Forgetting by Preserving Global Knowledge Gradients in Federated Learning with Non-IID Data
The inevitable presence of data heterogeneity has made federated learning very challenging. There are numerous methods to deal with this issue, such as local regularization, better model fusion techniques, and data shari…
Federated LearningTransfer LearningThe Computational Basis of Confidence in Large Language Models
Reliable confidence -- the probability that a model's own answer is correct -- is essential for the trustworthy deployment of language models. Existing work has largely evaluated confidence by how well it predicts correc…
Visual ReasoningInfluence Score and Transformers interpretability: Measure of the Effective Impact of Attention Heads at inference time
We propose an influence score to quantify the contribution of attention heads to classification decisions in Transformer-based models designed for prompt injection detection. The score combines directional influence on t…