On Calibrated Model Uncertainty in Deep Learning
Estimated uncertainty by approximate posteriors in Bayesian neural networks are prone to miscalibration, which leads to overconfident predictions in critical tasks that have a clear asymmetric cost or significant losses. Here, we extend the approximate inference for the loss-calibrated Bayesian framework to dropweights based Bayesian neural networks by maximising expected utility over a model posterior to calibrate uncertainty in deep learning. Furthermore, we show that decisions informed by loss-calibrated uncertainty can improve diagnostic performance to a greater extent than straightforward alternatives. We propose Maximum Uncertainty Calibration Error (MUCE) as a metric to measure calibrated confidence, in addition to its prediction especially for high-risk applications, where the goal is to minimise the worst-case deviation between error and estimated uncertainty. In experiments, we show the correlation between error in prediction and estimated uncertainty by interpreting Wasserstein distance as the accuracy of prediction. We evaluated the effectiveness of our approach to detecting Covid-19 from X-Ray images. Experimental results show that our method reduces miscalibration considerably, without impacting the models accuracy and improves reliability of computer-based diagnostics.
Code (0)
등록된 구현이 없습니다.
Tasks
Deep LearningDiagnosticmodelPredictionSimilar Papers 제목 키워드 기반
From Entropy to Calibrated Uncertainty: Training Language Models to Reason About Uncertainty
Large Language Models (LLMs) that can express interpretable and calibrated uncertainty are crucial in high-stakes domains. While methods to compute uncertainty post-hoc exist, they are often sampling-based and therefore …
Reinforcement LearningTeaching Models to Express Their Uncertainty in Words
We show that a GPT-3 model can learn to express uncertainty about its own answers in natural language -- without use of model logits. When given a question, the model generates both an answer and a level of confidence (e…
Calibrated Uncertainty for Molecular Property Prediction using Ensembles of Message Passing Neural Networks
Data-driven methods based on machine learning have the potential to accelerate computational analysis of atomic structures. In this context, reliable uncertainty estimates are important for assessing confidence in predic…
BIG-bench Machine LearningDecision MakingMolecular Property PredictionProperty PredictionFast Calibrated Explanations: Efficient and Uncertainty-Aware Explanations for Machine Learning Models
This paper introduces Fast Calibrated Explanations, a method designed for generating rapid, uncertainty-aware explanations for machine learning models. By incorporating perturbation techniques from ConformaSight - a glob…
Computational EfficiencyFeature ImportanceregressionUncertainty QuantificationLikelihood Annealing: Fast Calibrated Uncertainty for Regression
Recent advances in deep learning have shown that uncertainty estimation is becoming increasingly important in applications such as medical imaging, natural language processing, and autonomous systems. However, accurately…
DenoisingImage Super-ResolutionProperty Predictionregression+1