Diagnostic Uncertainty Calibration: Towards Reliable Machine Predictions in Medical Domain
We propose an evaluation framework for class probability estimates (CPEs) in the presence of label uncertainty, which is commonly observed as diagnosis disagreement between experts in the medical domain. We also formalize evaluation metrics for higher-order statistics, including inter-rater disagreement, to assess predictions on label uncertainty. Moreover, we propose a novel post-hoc method called $alpha$-calibration, that equips neural network classifiers with calibrated distributions over CPEs. Using synthetic experiments and a large-scale medical imaging application, we show that our approach significantly enhances the reliability of uncertainty estimates: disagreement probabilities and posterior CPEs.
Code (0)
등록된 구현이 없습니다.
Tasks
DiagnosticSimilar Papers 제목 키워드 기반
From Keypoints to Predictive Distributions: Post-Hoc Uncertainty for YOLO-Pose Models
YOLO-Pose models provide efficient keypoint localization, but do not quantify the associated spatial uncertainty. We introduce a lightweight post-hoc probabilistic extension that augments a trained YOLO-Pose model with c…
HypUC: Hyperfine Uncertainty Calibration with Gradient-boosted Corrections for Reliable Regression on Imbalanced Electrocardiograms
The automated analysis of medical time series, such as the electrocardiogram (ECG), electroencephalogram (EEG), pulse oximetry, etc, has the potential to serve as a valuable tool for diagnostic decisions, allowing for re…
DiagnosticEEGElectroencephalogram (EEG)regression+1Calibration in Machine Learning Uncertainty Quantification: beyond consistency to target adaptivity
Reliable uncertainty quantification (UQ) in machine learning (ML) regression tasks is becoming the focus of many studies in materials and chemical science. It is now well understood that average calibration is insufficie…
Uncertainty QuantificationEfficient Post-Hoc Uncertainty Calibration via Variance-Based Smoothing
Since state-of-the-art uncertainty estimation methods are often computationally demanding, we investigate whether incorporating prior information can improve uncertainty estimates in conventional deep neural networks. Ou…
Calibration of Encoder Decoder Models for Neural Machine Translation
We study the calibration of several state of the art neural machine translation(NMT) systems built on attention-based encoder-decoder models. For structured outputs like in NMT, calibration is important not just for reli…
DecoderMachine TranslationNMTTranslation