Bayesian posterior approximation with stochastic ensembles

2022-12-15 · CVPR 2023 1 · Oleksandr Balabanov, Bernhard Mehlig, Hampus Linander

We introduce ensembles of stochastic neural networks to approximate the Bayesian posterior, combining stochastic methods such as dropout with deep ensembles. The stochastic ensembles are formulated as families of distributions and trained to approximate the Bayesian posterior with variational inference. We implement stochastic ensembles based on Monte Carlo dropout, DropConnect and a novel non-parametric version of dropout and evaluate them on a toy problem and CIFAR image classification. For both tasks, we test the quality of the posteriors directly against Hamiltonian Monte Carlo simulations. Our results show that stochastic ensembles provide more accurate posterior estimates than other popular baselines for Bayesian inference.

📄 PDF Abstract BibTeX arXiv:2212.08123

Code (1)

oleksandr-balabanov/stochastic-ensembles 공식 구현 pytorch

Tasks

Bayesian Inference image-classification Image ClassificationVariational Inference

Methods 이 논문이 사용한 방법론

Test 설명 없음

DropConnect DropConnect generalizes Dropout by randomly dropping the weights rather than the activations with probability $1-p$. DropConnect…

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Scalable Bayesian Learning with posteriors

2024-05-31 · Samuel Duffield, Kaelan Donatella, Johnathan Chiu, Phoebe Klett 외

Although theoretically compelling, Bayesian learning with modern machine learning models is computationally challenging since it requires approximating a high dimensional posterior distribution. In this work, we (i) intr…

Efficient variational Bayesian neural network ensembles for outlier detection

2017-03-20 · Nick Pawlowski, Miguel Jaques, Ben Glocker

In this work we perform outlier detection using ensembles of neural networks obtained by variational approximation of the posterior in a Bayesian neural network setting. The variational parameters are obtained by samplin…

Outlier Detection

SAE: Sequential Anchored Ensembles

2021-12-30 · Arnaud Delaunoy, Gilles Louppe

Computing the Bayesian posterior of a neural network is a challenging task due to the high-dimensionality of the parameter space. Anchored ensembles approximate the posterior by training an ensemble of neural networks on…

Bayesian Deep Ensembles via the Neural Tangent Kernel

2020-07-11 · NeurIPS 2020 12 · Bobby He, Balaji Lakshminarayanan, Yee Whye Teh

We explore the link between deep ensembles and Gaussian processes (GPs) through the lens of the Neural Tangent Kernel (NTK): a recent development in understanding the training dynamics of wide neural networks (NNs). Prev…

Gaussian Processes

Greedy Bayesian Posterior Approximation with Deep Ensembles

2021-05-29 · Aleksei Tiulpin, Matthew B. Blaschko

Ensembles of independently trained neural networks are a state-of-the-art approach to estimate predictive uncertainty in Deep Learning, and can be interpreted as an approximation of the posterior distribution via a mixtu…

Diversity Out-of-Distribution Detection