Logit-Based Ensemble Distribution Distillation for Robust Autoregressive Sequence Uncertainties
Efficiently and reliably estimating uncertainty is an important objective in deep learning. It is especially pertinent to autoregressive sequence tasks, where training and inference costs are typically very high. However, existing research has predominantly focused on tasks with static data such as image classification. In this work, we investigate Ensemble Distribution Distillation (EDD) applied to large-scale natural language sequence-to-sequence data. EDD aims to compress the superior uncertainty performance of an expensive (teacher) ensemble into a cheaper (student) single model. Importantly, the ability to separate knowledge (epistemic) and data (aleatoric) uncertainty is retained. Existing probability-space approaches to EDD, however, are difficult to scale to large vocabularies. We show, for modern transformer architectures on large-scale translation tasks, that modelling the ensemble logits, instead of softmax probabilities, leads to significantly better students. Moreover, the students surprisingly even outperform Deep Ensembles by up to ~10% AUROC on out-of-distribution detection, whilst matching them at in-distribution translation.
Code (0)
등록된 구현이 없습니다.
Tasks
image-classificationImage ClassificationOut-of-Distribution DetectionTranslationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Logit Distance Bounds Representational Similarity
For a broad family of discriminative models that includes autoregressive language models, identifiability results imply that if two models induce the same conditional distributions, then their internal representations ar…
Ensemble Modeling with Contrastive Knowledge Distillation for Sequential Recommendation
Sequential recommendation aims to capture users' dynamic interest and predicts the next item of users' preference. Most sequential recommendation methods use a deep neural network as sequence encoder to generate user and…
AttributeContrastive LearningKnowledge DistillationMulti-Task Learning+2PILE: Pairwise Iterative Logits Ensemble for Multi-Teacher Labeled Distillation
Pre-trained language models have become a crucial part of ranking systems and achieved very impressive effects recently. To maintain high performance while keeping efficient computations, knowledge distillation is widely…
Knowledge DistillationELODI: Ensemble Logit Difference Inhibition for Positive-Congruent Training
Negative flips are errors introduced in a classification system when a legacy model is updated. Existing methods to reduce the negative flip rate (NFR) either do so at the expense of overall accuracy by forcing a new mod…
Classificationimage-classificationImage ClassificationData-Efficient Autoregressive-to-Diffusion Language Models via On-Policy Distillation
We study the transformation of autoregressive models (ARLMs) into diffusion language models (DLMs). Rather than pretraining from scratch, prior work replaces the causal attention in ARLMs with bidirectional attention and…