Uncertainty Calibration Error: A New Metric for Multi-Class Classification
Various metrics have recently been proposed to measure uncertainty calibration of deep models for classification. However, these metrics either fail to capture miscalibration correctly or lack interpretability. We propose to use the normalized entropy as a measure of uncertainty and derive the Uncertainty Calibration Error (UCE), a comprehensible calibration metric for multi-class classification. In our experiments, we focus on uncertainty from variational Bayesian inference methods and compare UCE to established calibration errors on the task of multi-class image classification. UCE avoids several pathologies of other metrics, but does not sacrifice interpretability. It can be used for regularization to improve calibration during training without penalizing predictions with justified high confidence.
Code (0)
등록된 구현이 없습니다.
Tasks
Bayesian InferenceClassificationGeneral Classificationimage-classificationImage ClassificationMulti-class ClassificationSimilar Papers 제목 키워드 기반
Inferring bias and uncertainty in camera calibration
Accurate camera calibration is a precondition for many computer vision applications. Calibration errors, such as wrong model assumptions or imprecise parameter estimation, can deteriorate a system's overall performance, …
Bias DetectionCamera CalibrationModel Selectionparameter estimationOn the Calibration of Probabilistic Classifier Sets
Multi-class classification methods that produce sets of probabilistic classifiers, such as ensemble learning methods, are able to model aleatoric and epistemic uncertainty. Aleatoric uncertainty is then typically quantif…
Ensemble LearningMulti-class ClassificationCan we trust our models? Epistemic calibration in second-order classification
Uncertainty estimation is critical for deploying machine learning models in high-stakes settings. However, classical calibration only assesses the reliability of predicted probabilities and does not evaluate whether epis…
Geometric Calibration and Neutral Zones for Uncertainty-Aware Multi-Class Classification
Modern artificial intelligence systems make critical decisions yet often fail silently when uncertain -- even well-calibrated models provide no mechanism to identify \textit{which specific predictions} are unreliable. We…
Multi-class ClassificationWhy Calibration Error is Wrong Given Model Uncertainty: Using Posterior Predictive Checks with Deep Learning
Within the last few years, there has been a move towards using statistical models in conjunction with neural networks with the end goal of being able to better answer the question, "what do our models know?". From this t…