Venn-Abers predictors
This paper continues study, both theoretical and empirical, of the method of Venn prediction, concentrating on binary prediction problems. Venn predictors produce probability-type predictions for the labels of test objects which are guaranteed to be well calibrated under the standard assumption that the observations are generated independently from the same distribution. We give a simple formalization and proof of this property. We also introduce Venn-Abers predictors, a new class of Venn predictors based on the idea of isotonic regression, and report promising empirical results both for Venn-Abers predictors and for their more computationally efficient simplified version.
Code (1)
Tasks
PredictionregressionSimilar Papers 제목 키워드 기반
Multi-class probabilistic classification using inductive and cross Venn-Abers predictors
Inductive (IVAP) and cross (CVAP) Venn–Abers predictors are computationally efficient algorithms for probabilistic prediction in binary classification problems. We present a new approach to multi-class probability estima…
Binary ClassificationClassificationClassifier calibrationConformal Prediction+1Well-Calibrated Probabilistic Predictive Maintenance using Venn-Abers
When using machine learning for fault detection, a common problem is the fact that most data sets are very unbalanced, with the minority class (a fault) being the interesting one. In this paper, we investigate the usage …
Fault DetectionvalidInductive Venn-Abers and related regressors
Venn-Abers predictors are probabilistic predictors that enjoy appealing properties of validity, but their major limitation is that they have been applicable only to binary classification, apart from a recent extension to…
Binary ClassificationCalibration of Natural Language Understanding Models with Venn--ABERS Predictors
Transformers, currently the state-of-the-art in natural language understanding (NLU) tasks, are prone to generate uncalibrated predictions or extreme probabilities, making the process of taking different decisions based …
Natural Language UnderstandingClassifier Calibration at Scale: An Empirical Study of Model-Agnostic Post-Hoc Methods
We study model-agnostic post-hoc calibration methods intended to improve probabilistic predictions in supervised binary classification on real i.i.d. tabular data, with particular emphasis on conformal and Venn-based app…
Classifier calibrationBinary Classification