Uncertainty Quantification Metrics for Deep Regression
When deploying deep neural networks on robots or other physical systems, the learned model should reliably quantify predictive uncertainty. A reliable uncertainty allows downstream modules to reason about the safety of its actions. In this work, we address metrics for evaluating such an uncertainty. Specifically, we focus on regression tasks, and investigate Area Under Sparsification Error (AUSE), Calibration Error, Spearman's Rank Correlation, and Negative Log-Likelihood (NLL). Using synthetic regression datasets, we look into how those metrics behave under four typical types of uncertainty, their stability regarding the size of the test set, and reveal their strengths and weaknesses. Our results indicate that Calibration Error is the most stable and interpretable metric, but AUSE and NLL also have their respective use cases. We discourage the usage of Spearman's Rank Correlation for evaluating uncertainties and recommend replacing it with AUSE.
Code (0)
등록된 구현이 없습니다.
Tasks
regressionUncertainty QuantificationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Empirical Frequentist Coverage of Deep Learning Uncertainty Quantification Procedures
Uncertainty quantification for complex deep learning models is increasingly important as these techniques see growing use in high-stakes, real-world settings. Currently, the quality of a model's uncertainty is evaluated …
Deep LearningUncertainty QuantificationA framework for benchmarking uncertainty in deep regression
We propose a framework for the assessment of uncertainty quantification in deep regression. The framework is based on regression problems where the regression function is a linear combination of nonlinear functions. Basi…
BenchmarkingregressionUncertainty QuantificationBetter Uncertainty Quantification for Machine Translation Evaluation
Neural-based machine translation (MT) evaluation metrics are progressing fast. However, they are often hard to interpret and might produce unreliable scores when human references or assessments are noisy or when data is …
Machine TranslationTranslationUncertainty QuantificationGradient-Based Quantification of Epistemic Uncertainty for Deep Object Detectors
The vast majority of uncertainty quantification methods for deep object detectors such as variational inference are based on the network output. Here, we study gradient-based epistemic uncertainty metrics for deep object…
Objectobject-detectionObject DetectionUncertainty Quantification+1An Axiomatic Assessment of Entropy- and Variance-based Uncertainty Quantification in Regression
Uncertainty quantification (UQ) is crucial in machine learning, yet most (axiomatic) studies of uncertainty measures focus on classification, leaving a gap in regression settings with limited formal justification and eva…
regressionUncertainty Quantification