Calibration tests in multi-class classification: A unifying framework
In safety-critical applications a probabilistic model is usually required to be calibrated, i.e., to capture the uncertainty of its predictions accurately. In multi-class classification, calibration of the most confident predictions only is often not sufficient. We propose and study calibration measures for multi-class classification that generalize existing measures such as the expected calibration error, the maximum calibration error, and the maximum mean calibration error. We propose and evaluate empirically different consistent and unbiased estimators for a specific class of measures based on matrix-valued kernels. Importantly, these estimators can be interpreted as test statistics associated with well-defined bounds and approximations of the p-value under the null hypothesis that the model is calibrated, significantly improving the interpretability of calibration measures, which otherwise lack any meaningful unit or scale.
Code (1)
Tasks
ClassificationGeneral ClassificationMulti-class ClassificationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Calibration tests beyond classification
Most supervised machine learning tasks are subject to irreducible prediction errors. Probabilistic predictive models address this limitation by providing probability distributions that represent a belief over plausible t…
ClassificationDecision MakingDecision Making Under UncertaintyMulti-class Classification+1Socrates Loss: Unifying Confidence Calibration and Classification by Leveraging the Unknown
Deep neural networks, despite their high accuracy, often exhibit poor confidence calibration, limiting their reliability in high-stakes applications. Current ad-hoc confidence calibration methods attempt to fix this duri…
A Unifying Perspective on Multi-Calibration: Game Dynamics for Multi-Objective Learning
We provide a unifying framework for the design and analysis of multicalibrated predictors. By placing the multicalibration problem in the general setting of multi-objective learning -- where learning guarantees must hold…
FairnessOn Calibrating Semantic Segmentation Models: Analyses and An Algorithm
We study the problem of semantic segmentation calibration. Lots of solutions have been proposed to approach model miscalibration of confidence in image classification. However, to date, confidence calibration research on…
image-classificationImage ClassificationPredictionSegmentation+1Calibrated and uncertain? Evaluating uncertainty estimates in binary classification models
Rigorous statistical methods, including parameter estimation with accompanying uncertainties, underpin the validity of scientific discovery, especially in the natural sciences. With increasingly complex data models such …
Binary ClassificationBayesian Inference