On Batch Normalisation for Approximate Bayesian Inference
We study batch normalisation in the context of variational inference methods in Bayesian neural networks, such as mean-field or MC Dropout. We show that batch-normalisation does not affect the optimum of the evidence lower bound (ELBO). Furthermore, we study the Monte Carlo Batch Normalisation (MCBN) algorithm, proposed as an approximate inference technique parallel to MC Dropout, and show that for larger batch sizes, MCBN fails to capture epistemic uncertainty. Finally, we provide insights into what is required to fix this failure, namely having to view the mini-batch size as a variational parameter in MCBN. We comment on the asymptotics of the ELBO with respect to this variational parameter, showing that as dataset size increases towards infinity, the batch-size must increase towards infinity as well for MCBN to be a valid approximate inference technique.
Code (0)
등록된 구현이 없습니다.
Tasks
Bayesian InferencevalidVariational InferenceMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Partial transfusion: on the expressive influence of trainable batch norm parameters for transfer learning
Transfer learning from ImageNet is the go-to approach when applying deep learning to medical images. The approach is either to fine-tune a pre-trained model or use it as a feature extractor. Most modern architecture cont…
Transfer LearningLinearised Laplace Inference in Networks with Normalisation Layers and the Neural g-Prior
We show that for neural networks (NN) with normalisation layers, i.e. batch norm, layer norm, or group norm, the Laplace model evidence does not approximate the volume of a posterior mode and is thus unsuitable for model…
image-classificationImage ClassificationModel SelectionregressionPractical Deep Learning with Bayesian Principles
Bayesian methods promise to fix many shortcomings of deep learning, but they are impractical and rarely match the performance of standard methods, let alone improve them. In this paper, we demonstrate practical training …
Continual LearningData AugmentationDeep LearningVariational InferenceHybrid Batch Normalisation: Resolving the Dilemma of Batch Normalisation in Federated Learning
Batch Normalisation (BN) is widely used in conventional deep neural network training to harmonise the input-output distributions for each batch of data. However, federated learning, a distributed learning paradigm, faces…
Federated LearningRobust Generalised Bayesian Inference for Intractable Likelihoods
Generalised Bayesian inference updates prior beliefs using a loss function, rather than a likelihood, and can therefore be used to confer robustness against possible mis-specification of the likelihood. Here we consider …
Bayesian Inference