A Variational View on Bootstrap Ensembles as Bayesian Inference
In this paper, we employ variational arguments to establish a connection between ensemble methods for Neural Networks and Bayesian inference. We consider an ensemble-based scheme where each model/particle corresponds to a perturbation of the data by means of parametric bootstrap and a perturbation of the prior. We derive conditions under which any optimization steps of the particles makes the associated distribution reduce its divergence to the posterior over model parameters. Such conditions do not require any particular form for the approximation and they are purely geometrical, giving insights on the behavior of the ensemble on a number of interesting models such as Neural Networks with ReLU activations. Experiments confirm that ensemble methods can be a valid alternative to approximate Bayesian inference; the theoretical developments in the paper seek to explain this behavior.
Code (0)
등록된 구현이 없습니다.
Tasks
Bayesian InferencevalidMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Parametric Bootstrap Ensembles as Variational Inference
In this paper, we employ variational arguments to establish a connection between ensemble methods for Neural Networks and Bayesian inference. We consider an ensemble-based scheme where each model/particle corresponds to …
Bayesian InferenceVariational InferenceA Rigorous Link between Deep Ensembles and (Variational) Bayesian Methods
We establish the first mathematically rigorous link between Bayesian, variational Bayesian, and ensemble methods. A key step towards this it to reformulate the non-convex optimisation problem typically encountered in dee…
Deep LearningUncertainty QuantificationVariational InferenceDeep Ensembles: A Loss Landscape Perspective
Deep ensembles have been empirically shown to be a promising approach for improving accuracy, uncertainty and out-of-distribution robustness of deep learning models. While deep ensembles were theoretically motivated by t…
Bayesian posterior approximation with stochastic ensembles
We introduce ensembles of stochastic neural networks to approximate the Bayesian posterior, combining stochastic methods such as dropout with deep ensembles. The stochastic ensembles are formulated as families of distrib…
Bayesian Inferenceimage-classificationImage ClassificationVariational InferenceOn the use of bootstrap with variational inference: Theory, interpretation, and a two-sample test example
Variational inference is a general approach for approximating complex density functions, such as those arising in latent variable models, popular in machine learning. It has been applied to approximate the maximum likeli…
Bayesian InferenceVariational Inference