The Peril of Popular Deep Learning Uncertainty Estimation Methods

2021-12-09 · Yehao Liu, Matteo Pagliardini, Tatjana Chavdarova, Sebastian U. Stich

Uncertainty estimation (UE) techniques -- such as the Gaussian process (GP), Bayesian neural networks (BNN), Monte Carlo dropout (MCDropout) -- aim to improve the interpretability of machine learning models by assigning an estimated uncertainty value to each of their prediction outputs. However, since too high uncertainty estimates can have fatal consequences in practice, this paper analyzes the above techniques. Firstly, we show that GP methods always yield high uncertainty estimates on out of distribution (OOD) data. Secondly, we show on a 2D toy example that both BNNs and MCDropout do not give high uncertainty estimates on OOD samples. Finally, we show empirically that this pitfall of BNNs and MCDropout holds on real world datasets as well. Our insights (i) raise awareness for the more cautious use of currently popular UE methods in Deep Learning, (ii) encourage the development of UE methods that approximate GP-based methods -- instead of BNNs and MCDropout, and (iii) our empirical setups can be used for verifying the OOD performances of any other UE method. The source code is available at https://github.com/epfml/uncertainity-estimation.

📄 PDF Abstract BibTeX arXiv:2112.05000

Code (1)

epfml/uncertainity-estimation 공식 구현

Tasks

Deep Learning

Methods 이 논문이 사용한 방법론

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Gaussian Process Gaussian Processes are non-parametric models for approximating functions. They rely upon a measure of similarity between points (the kernel function) to predict the value for…

Monte Carlo Dropout 설명 없음

PERIL: Probabilistic Embeddings for hybrid Meta-Reinforcement and Imitation Learning

2021-01-01 · Alvaro Prat, Edward Johns

Imitation learning is a natural way for a human to describe a task to an agent, and it can be combined with reinforcement learning to enable the agent to solve that task through exploration. However, traditional methods …

Imitation Learning Meta Reinforcement Learning reinforcement-learning Reinforcement Learning+1

An Uncertainty Estimation Framework for Probabilistic Object Detection

2021-06-28 · Zongyao Lyu, Nolan B. Gutierrez, William J. Beksi

In this paper, we introduce a new technique that combines two popular methods to estimate uncertainty in object detection. Quantifying uncertainty is critical in real-world robotic applications. Traditional detection mod…

Object object-detection Object Detection

Temporal Distribution Shift in Real-World Pharmaceutical Data: Implications for Uncertainty Quantification in QSAR Models

2025-02-06 · Hannah Rosa Friesacher, Emma Svensson, Susanne Winiwarter, Lewis Mervin 외

The estimation of uncertainties associated with predictions from quantitative structure-activity relationship (QSAR) models can accelerate the drug discovery process by identifying promising experiments and allowing an e…

Drug Discovery Uncertainty Quantification

Which models are innately best at uncertainty estimation?

2022-06-05 · Ido Galil, Mohammed Dabbah, Ran El-Yaniv

Due to the comprehensive nature of this paper, it has been updated and split into two separate papers: "A Framework For Benchmarking Class-out-of-distribution Detection And Its Application To ImageNet" and "What Can We L…

Benchmarking Out-of-Distribution Detection

What Can We Learn From The Selective Prediction And Uncertainty Estimation Performance Of 523 Imagenet Classifiers

2023-02-23 · Ido Galil, Mohammed Dabbah, Ran El-Yaniv

When deployed for risk-sensitive tasks, deep neural networks must include an uncertainty estimation mechanism. Here we examine the relationship between deep architectures and their respective training regimes, with their…