The Art of Abstention: Selective Prediction and Error Regularization for Natural Language Processing
In selective prediction, a classifier is allowed to abstain from making predictions on low-confidence examples. Though this setting is interesting and important, selective prediction has rarely been examined in natural language processing (NLP) tasks. To fill this void in the literature, we study in this paper selective prediction for NLP, comparing different models and confidence estimators. We further propose a simple error regularization trick that improves confidence estimation without substantially increasing the computation budget. We show that recent pre-trained transformer models simultaneously improve both model accuracy and confidence estimation effectiveness. We also find that our proposed regularization improves confidence estimation and can be applied to other relevant scenarios, such as using classifier cascades for accuracy{--}efficiency trade-offs. Source code for this paper can be found at https://github.com/castorini/transformers-selective.
Code (1)
Tasks
PredictionSimilar Papers 제목 키워드 기반
Selective Classification Can Magnify Disparities Across Groups
Selective classification, in which models can abstain on uncertain predictions, is a natural approach to improving accuracy in settings where errors are costly but abstentions are manageable. In this paper, we find that …
ClassificationGeneral ClassificationSelective "Selective Prediction": Reducing Unnecessary Abstention in Vision-Language Reasoning
Selective prediction minimizes incorrect predictions from vision-language models (VLMs) by allowing them to abstain from answering when uncertain. However, when deploying a vision-language system with low tolerance for i…
PredictionSelective Nonparametric Regression via Testing
Prediction with the possibility of abstention (or selective prediction) is an important problem for error-critical machine learning applications. While well-studied in the classification setup, selective approaches to re…
regressionPost-Abstention: Towards Reliably Re-Attempting the Abstained Instances in QA
Despite remarkable progress made in natural language processing, even the state-of-the-art models often make incorrect predictions. Such predictions hamper the reliability of systems and limit their widespread adoption i…
Explicit Abstention Knobs for Predictable Reliability in Video Question Answering
High-stakes deployment of vision-language models (VLMs) requires selective prediction, where systems abstain when uncertain rather than risk costly errors. We investigate whether confidence-based abstention provides reli…
Video Question Answering