One Simple Trick to Fix Your Bayesian Neural Network
One of the most popular estimation methods in Bayesian neural networks (BNN) is mean-field variational inference (MFVI). In this work, we show that neural networks with ReLU activation function induce posteriors, that are hard to fit with MFVI. We provide a theoretical justification for this phenomenon, study it empirically, and report the results of a series of experiments to investigate the effect of activation function on the calibration of BNNs. We find that using Leaky ReLU activations leads to more Gaussian-like weight posteriors and achieves a lower expected calibration error (ECE) than its ReLU-based counterpart.
Code (0)
등록된 구현이 없습니다.
Tasks
Variational InferenceMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
ScrofaZero: Mastering Trick-taking Poker Game Gongzhu by Deep Reinforcement Learning
People have made remarkable progress in game AIs, especially in domain of perfect information game. However, trick-taking poker game, as a popular form of imperfect information game, has been regarded as a challenge for …
Bayesian InferenceDeep Reinforcement Learningreinforcement-learningReinforcement Learning (RL)Optimize Your Sampling: Tuned Diffusion Sampling with Bayesian Optimization
Sampling from a diffusion model typically requires many forward passes through a large neural network, making generation computationally expensive. While much work has focused on efficient solvers and samplers, comparati…
Text-to-Image GenerationBayesian statistical learning using density operators
This short study reformulates the statistical Bayesian learning problem using a quantum mechanics framework. Density operators representing ensembles of pure states of sample wave functions are used in place probability …
A surprisingly simple technique to control the pretraining bias for better transfer: Expand or Narrow your representation
Self-Supervised Learning (SSL) models rely on a pretext task to learn representations. Because this pretext task differs from the downstream tasks used to evaluate the performance of these models, there is an inherent mi…
Self-Supervised LearningAll your loss are belong to Bayes
Loss functions are a cornerstone of machine learning and the starting point of most algorithms. Statistics and Bayesian decision theory have contributed, via properness, to elicit over the past decades a wide set of admi…
AllGaussian Processes