Uncertainty Quantification for Machine Learning: One Size Does Not Fit All
Proper quantification of predictive uncertainty is essential for the use of machine learning in safety-critical applications. Various uncertainty measures have been proposed for this purpose, typically claiming superiority over other measures. In this paper, we argue that there is no single best measure. Instead, uncertainty quantification should be tailored to the specific application. To this end, we use a flexible family of uncertainty measures that distinguishes between total, aleatoric, and epistemic uncertainty of second-order distributions. These measures can be instantiated with specific loss functions, so-called proper scoring rules, to control their characteristics, and we show that different characteristics are useful for different tasks. In particular, we show that, for the task of selective prediction, the scoring rule should ideally match the task loss. On the other hand, for out-of-distribution detection, our results confirm that mutual information, a widely used measure of epistemic uncertainty, performs best. Furthermore, in an active learning setting, epistemic uncertainty based on zero-one loss is shown to consistently outperform other uncertainty measures.
Code (0)
등록된 구현이 없습니다.
Tasks
Out-of-Distribution DetectionActive LearningSimilar Papers 제목 키워드 기반
Old Rules in a New Game: Mapping Uncertainty Quantification to Quantum Machine Learning
One of the key obstacles in traditional deep learning is the reduction in model transparency caused by increasingly intricate model functions, which can lead to problems such as overfitting and excessive confidence in pr…
Quantum Machine LearningUncertainty Quantification for Transformer Models for Dark-Pattern Detection
The opaque nature of transformer-based models, particularly in applications susceptible to unethical practices such as dark-patterns in user interfaces, requires models that integrate uncertainty quantification to enhanc…
Gaussian ProcessesUncertainty QuantificationApproximate Bayesian Neural Operators: Uncertainty Quantification for Parametric PDEs
Neural operators are a type of deep architecture that learns to solve (i.e. learns the nonlinear solution operator of) partial differential equations (PDEs). The current state of the art for these models does not provide…
Gaussian ProcessesUncertainty QuantificationPitfalls of Epistemic Uncertainty Quantification through Loss Minimisation
Uncertainty quantification has received increasing attention in machine learning in the recent past. In particular, a distinction between aleatoric and epistemic uncertainty has been found useful in this regard. The latt…
Uncertainty QuantificationUncertainty quantification for improving radiomic-based models in radiation pneumonitis prediction
Background: Radiation pneumonitis is a side effect of thoracic radiation therapy. Recently, machine learning models with radiomic features have improved radiation pneumonitis prediction by capturing spatial information. …
Conformal PredictionDecision MakingPredictionregression+1