Joint Training for Selective Prediction
Classifier models are prevalent in natural language processing (NLP), often with high accuracy. Yet in real world settings, human-in-the-loop systems can foster trust in model outputs and even higher performance. Selective Prediction (SP) methods determine when to adopt a classifier's output versus defer to a human. Previous SP approaches have addressed how to improve softmax as a measure of model confidence, or have developed separate confidence estimators. One previous method involves learning a deferral model based on engineered features. We introduce a novel joint-training approach that simultaneously optimizes learned representations used by the classifier module and a learned deferral policy. Our results on four classification tasks demonstrate that joint training not only leads to better SP outcomes over two strong baselines, but also improves the performance of both modules.
Code (0)
등록된 구현이 없습니다.
Tasks
PredictionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Unsupervised Selective Rationalization with Noise Injection
A major issue with using deep learning models in sensitive applications is that they provide no explanation for their output. To address this problem, unsupervised selective rationalization produces rationales alongside …
Selective transfer learning with adversarial training for stock movement prediction
Stock movement prediction is a critical issue in the field of financial investment. It is very challenging since a stock usually shows highly stochastic property in price and has complex relationships with other stocks. …
PredictionStock PredictionTransfer LearningSelective Zero-Shot Classification with Augmented Attributes
In this paper, we introduce a selective zero-shot classification problem: how can the classifier avoid making dubious predictions? Existing attribute-based zero-shot classification methods are shown to work poorly in the…
AttributeClassificationGeneral Classificationzero-shot-classification+1CREST: A Joint Framework for Rationalization and Counterfactual Text Generation
Selective rationales and counterfactual examples have emerged as two effective, complementary classes of interpretability methods for analyzing and training NLP models. However, prior work has not explored how these meth…
counterfactualData AugmentationText GenerationvalidBeyond the Training Distribution: Evaluating Predictions Under Distribution Shift and Selection Bias
Understanding how a prediction model will perform in a new environment before deployment is essential to preventing harm when algorithms inform decision-making. Two common sources of model performance degradation are (i)…