Empirical Frequentist Coverage of Deep Learning Uncertainty Quantification Procedures
Uncertainty quantification for complex deep learning models is increasingly important as these techniques see growing use in high-stakes, real-world settings. Currently, the quality of a model's uncertainty is evaluated using point-prediction metrics such as negative log-likelihood or the Brier score on heldout data. In this study, we provide the first large scale evaluation of the empirical frequentist coverage properties of well known uncertainty quantification techniques on a suite of regression and classification tasks. We find that, in general, some methods do achieve desirable coverage properties on in distribution samples, but that coverage is not maintained on out-of-distribution data. Our results demonstrate the failings of current uncertainty quantification techniques as dataset shift increases and establish coverage as an important metric in developing models for real-world applications.
Code (1)
Tasks
Deep LearningUncertainty QuantificationSimilar Papers 제목 키워드 기반
Pointwise uncertainty quantification for sparse variational Gaussian process regression with a Brownian motion prior
We study pointwise estimation and uncertainty quantification for a sparse variational Gaussian process method with eigenvector inducing variables. For a rescaled Brownian motion prior, we derive theoretical guarantees an…
Uncertainty QuantificationLocally Valid and Discriminative Prediction Intervals for Deep Learning Models
Crucial for building trust in deep learning models for critical real-world applications is efficient and theoretically sound uncertainty quantification, a task that continues to be challenging. Useful uncertainty informa…
Conformal PredictionDeep LearningPredictionPrediction Intervals+2Uncertainty in Physics and AI: Taxonomy, Quantification, and Validation
Reliable uncertainty quantification is essential for the use of machine learning in physics, where scientific discoveries depend on validated probabilistic statements. We provide a structured overview of uncertainty quan…
Uncertainty quantification using martingales for misspecified Gaussian processes
We address uncertainty quantification for Gaussian processes (GPs) under misspecified priors, with an eye towards Bayesian Optimization (BO). GPs are widely used in BO because they easily enable exploration based on post…
Bayesian OptimizationGaussian ProcessesUncertainty QuantificationvalidGenerative Parameter Sampler For Scalable Uncertainty Quantification
Uncertainty quantification has been a core of the statistical machine learning, but its computational bottleneck has been a serious challenge for both Bayesians and frequentists. We propose a model-based framework in qua…
General ClassificationUncertainty Quantification