Uncertainty Estimation via Stochastic Batch Normalization
In this work, we investigate Batch Normalization technique and propose its probabilistic interpretation. We propose a probabilistic model and show that Batch Normalization maximazes the lower bound of its marginalized log-likelihood. Then, according to the new probabilistic model, we design an algorithm which acts consistently during train and test. However, inference becomes computationally inefficient. To reduce memory and computational cost, we propose Stochastic Batch Normalization -- an efficient approximation of proper inference procedure. This method provides us with a scalable uncertainty estimation technique. We demonstrate the performance of Stochastic Batch Normalization on popular architectures (including deep convolutional architectures: VGG-like and ResNets) for MNIST and CIFAR-10 datasets.
Code (1)
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Stochastic Normalizations as Bayesian Learning
In this work we investigate the reasons why Batch Normalization (BN) improves the generalization performance of deep networks. We argue that one major reason, distinguishing it from data-independent normalization methods…
Population Normalization for Federated Learning
Batch normalization (BN) is widely recognized as an essential method in training deep neural networks, facilitating convergence and enhancing model stability. However, in Federated Learning (FL) contexts, where train…
Federated LearningFast Bayesian Uncertainty Estimation and Reduction of Batch Normalized Single Image Super-Resolution Network
Convolutional neural network (CNN) has achieved unprecedented success in image super-resolution tasks in recent years. However, the network's performance depends on the distribution of the training sets and degrades on o…
Adversarial DefenseImage ReconstructionImage Super-ResolutionSuper-ResolutionIterative Normalization: Beyond Standardization towards Efficient Whitening
Batch Normalization (BN) is ubiquitously employed for accelerating neural network training and improving the generalization capability by performing standardization within mini-batches. Decorrelated Batch Normalization (…
Robust Object DetectionNeural Sampling Machine with Stochastic Synapse allows Brain-like Learning and Inference
Many real-world mission-critical applications require continual online learning from noisy data and real-time decision making with a defined confidence level. Probabilistic models and stochastic neural networks can expli…
Bayesian InferenceDecision Makingimage-classificationImage Classification