Evaluating and Boosting Uncertainty Quantification in Classification
Emergence of artificial intelligence techniques in biomedical applications urges the researchers to pay more attention on the uncertainty quantification (UQ) in machine-assisted medical decision making. For classification tasks, prior studies on UQ are difficult to compare with each other, due to the lack of a unified quantitative evaluation metric. Considering that well-performing UQ models ought to know when the classification models act incorrectly, we design a new evaluation metric, area under Confidence-Classification Characteristic curves (AUCCC), to quantitatively evaluate the performance of the UQ models. AUCCC is threshold-free, robust to perturbation, and insensitive to the classification performance. We evaluate several UQ methods (e.g., max softmax output) with AUCCC to validate its effectiveness. Furthermore, a simple scheme, named Uncertainty Distillation (UDist), is developed to boost the UQ performance, where a confidence model is distilling the confidence estimated by deep ensembles. The proposed method is easy to implement; it consistently outperforms strong baselines on natural and medical image datasets in our experiments.
Code (0)
등록된 구현이 없습니다.
Tasks
ClassificationDecision MakingGeneral ClassificationUncertainty QuantificationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Combating Noise: Semi-supervised Learning by Region Uncertainty Quantification
Semi-supervised learning aims to leverage a large amount of unlabeled data for performance boosting. Existing works primarily focus on image classification. In this paper, we delve into semi-supervised learning for objec…
image-classificationImage Classificationobject-detectionObject Detection+1Uncertainty Quantification With Noise Injection in Neural Networks: A Bayesian Perspective
Model uncertainty quantification involves measuring and evaluating the uncertainty linked to a model's predictions, helping assess their reliability and confidence. Noise injection is a technique used to enhance the robu…
Bayesian InferenceUncertainty QuantificationA Novel Framework for Uncertainty Quantification via Proper Scores for Classification and Beyond
In this PhD thesis, we propose a novel framework for uncertainty quantification in machine learning, which is based on proper scores. Uncertainty quantification is an important cornerstone for trustworthy and reliable ma…
Uncertainty quantification for improving radiomic-based models in radiation pneumonitis prediction
Background: Radiation pneumonitis is a side effect of thoracic radiation therapy. Recently, machine learning models with radiomic features have improved radiation pneumonitis prediction by capturing spatial information. …
Conformal PredictionDecision MakingPredictionregression+1Wasserstein Gradient Boosting: A Framework for Distribution-Valued Supervised Learning
Gradient boosting is a sequential ensemble method that fits a new weaker learner to pseudo residuals at each iteration. We propose Wasserstein gradient boosting, a novel extension of gradient boosting that fits a new wea…
regressionUncertainty Quantification