Efficient Low Rank Gaussian Variational Inference for Neural Networks
Bayesian neural networks are enjoying a renaissance driven in part by recent advances in variational inference (VI). The most common form of VI employs a fully factorized or mean-field distribution, but this is known to suffer from several pathologies, especially as we expect posterior distributions with highly correlated parameters. Current algorithms that capture these correlations with a Gaussian approximating family are difficult to scale to large models due to computational costs and high variance of gradient updates. By using a new form of the reparametrization trick, we derive a computationally efficient algorithm for performing VI with a Gaussian family with a low-rank plus diagonal covariance structure. We scale to deep feed-forward and convolutional architectures. We find that adding low-rank terms to parametrized diagonal covariance does not improve predictive performance except on small networks, but low-rank terms added to a constant diagonal covariance improves performance on small and large-scale network architectures.
Code (1)
Tasks
Variational InferenceSimilar Papers 제목 키워드 기반
Fast Low-Rank Bayesian Matrix Completion with Hierarchical Gaussian Prior Models
The problem of low rank matrix completion is considered in this paper. To exploit the underlying low-rank structure of the data matrix, we propose a hierarchical Gaussian prior model, where columns of the low-rank matrix…
Bayesian InferenceLow-Rank Matrix CompletionMatrix CompletionOn the Parameterization of Gaussian Mean Field Posteriors in Bayesian Neural Networks
Variational Bayesian Inference is a popular methodology for approximating posterior distributions in Bayesian neural networks. Recent work developing this class of methods has explored ever richer parameterizations of th…
Bayesian InferenceVariational InferenceThe k-tied Normal Distribution: A Compact Parameterization of Gaussian Mean Field Posteriors in Bayesian Neural Networks
Variational Bayesian Inference is a popular methodology for approximating posterior distributions over Bayesian neural network weights. Recent work developing this class of methods has explored ever richer parameterizati…
Bayesian InferenceVariational InferenceStatistical and Computational Trade-offs in Variational Inference: A Case Study in Inferential Model Selection
Variational inference has recently emerged as a popular alternative to the classical Markov chain Monte Carlo (MCMC) in large-scale Bayesian inference. The core idea is to trade statistical accuracy for computational eff…
Bayesian InferenceComputational EfficiencyModel SelectionStochastic Optimization+2Amortized Variational Inference for Simple Hierarchical Models
It is difficult to use subsampling with variational inference in hierarchical models since the number of local latent variables scales with the dataset. Thus, inference in hierarchical models remains a challenge at large…
Variational Inference