On the Convergence of Stochastic Variational Inference in Bayesian Networks
We highlight a pitfall when applying stochastic variational inference to general Bayesian networks. For global random variables approximated by an exponential family distribution, natural gradient steps, commonly starting from a unit length step size, are averaged to convergence. This useful insight into the scaling of initial step sizes is lost when the approximation factorizes across a general Bayesian network, and care must be taken to ensure practical convergence. We experimentally investigate how much of the baby (well-scaled steps) is thrown out with the bath water (exact gradients).
Code (0)
등록된 구현이 없습니다.
Tasks
Variational InferenceSimilar Papers 제목 키워드 기반
Good Initializations of Variational Bayes for Deep Models
Stochastic variational inference is an established way to carry out approximate Bayesian inference for deep models. While there have been effective proposals for good initializations for loss minimization in deep learnin…
Bayesian InferenceGeneral ClassificationregressionVariational InferenceStochastic Variational Inference for Bayesian Sparse Gaussian Process Regression
This paper presents a novel variational inference framework for deriving a family of Bayesian sparse Gaussian process regression (SGPR) models whose approximations are variationally optimal with respect to the full-rank …
GPRregressionStochastic OptimizationVariational InferenceStochastic Collapsed Variational Bayesian Inference for Latent Dirichlet Allocation
In the internet era there has been an explosion in the amount of digital text information available, leading to difficulties of scale for traditional inference algorithms for topic models. Recent advances in stochastic v…
Bayesian InferenceTopic ModelsVariational InferenceOn the Parameterization of Gaussian Mean Field Posteriors in Bayesian Neural Networks
Variational Bayesian Inference is a popular methodology for approximating posterior distributions in Bayesian neural networks. Recent work developing this class of methods has explored ever richer parameterizations of th…
Bayesian InferenceVariational InferenceThe k-tied Normal Distribution: A Compact Parameterization of Gaussian Mean Field Posteriors in Bayesian Neural Networks
Variational Bayesian Inference is a popular methodology for approximating posterior distributions over Bayesian neural network weights. Recent work developing this class of methods has explored ever richer parameterizati…
Bayesian InferenceVariational Inference