Convergence Rates of Variational Inference in Sparse Deep Learning
Variational inference is becoming more and more popular for approximating intractable posterior distributions in Bayesian statistics and machine learning. Meanwhile, a few recent works have provided theoretical justification and new insights on deep neural networks for estimating smooth functions in usual settings such as nonparametric regression. In this paper, we show that variational inference for sparse deep learning retains the same generalization properties than exact Bayesian inference. In particular, we highlight the connection between estimation and approximation theories via the classical bias-variance trade-off and show that it leads to near-minimax rates of convergence for H\"older smooth functions. Additionally, we show that the model selection framework over the neural network architecture via ELBO maximization does not overfit and adaptively achieves the optimal rate of convergence.
Code (0)
등록된 구현이 없습니다.
Tasks
Bayesian InferenceDeep LearningModel SelectionVariational InferenceSimilar Papers 제목 키워드 기반
Convergence Rates of Empirical Bayes Posterior Distributions: A Variational Perspective
We study the convergence rates of empirical Bayes posterior distributions for nonparametric and high-dimensional inference. We show that as long as the hyperparameter set is discrete, the empirical Bayes posterior distri…
Density EstimationSpike and slab variational Bayes for high dimensional logistic regression
Variational Bayes (VB) is a popular scalable alternative to Markov chain Monte Carlo for Bayesian inference. We study a mean-field spike and slab VB approximation of widely used Bayesian model selection priors in sparse …
Bayesian InferenceModel SelectionregressionVocal Bursts Intensity PredictionConvergence Rates of Variational Posterior Distributions
We study convergence rates of variational posterior distributions for nonparametric and high-dimensional inference. We formulate general conditions on prior, likelihood, and variational class that characterize the conver…
On the Convergence of Black-Box Variational Inference
We provide the first convergence guarantee for full black-box variational inference (BBVI), also known as Monte Carlo variational inference. While preliminary investigations worked on simplified versions of BBVI (e.g., b…
Bayesian InferenceVariational InferenceStochastic Variational Inference for Bayesian Sparse Gaussian Process Regression
This paper presents a novel variational inference framework for deriving a family of Bayesian sparse Gaussian process regression (SGPR) models whose approximations are variationally optimal with respect to the full-rank …
GPRregressionStochastic OptimizationVariational Inference