A deterministic and computable Bernstein-von Mises theorem
Bernstein-von Mises results (BvM) establish that the Laplace approximation is asymptotically correct in the large-data limit. However, these results are inappropriate for computational purposes since they only hold over most, and not all, datasets and involve hard-to-estimate constants. In this article, I present a new BvM theorem which bounds the Kullback-Leibler (KL) divergence between a fixed log-concave density $f\left(\boldsymbol{\theta}\right)$ and its Laplace approximation. The bound goes to $0$ as the higher-derivatives of $f\left(\boldsymbol{\theta}\right)$ tend to $0$ and $f\left(\boldsymbol{\theta}\right)$ becomes increasingly Gaussian. The classical BvM theorem in the IID large-data asymptote is recovered as a corollary. Critically, this theorem further suggests a number of computable approximations of the KL divergence with the most promising being: \[ KL\left(g_{LAP},f\right)\approx\frac{1}{2}\text{Var}_{\boldsymbol{\theta}\sim g\left(\boldsymbol{\theta}\right)}\left(\log\left[f\left(\boldsymbol{\theta}\right)\right]-\log\left[g_{LAP}\left(\boldsymbol{\theta}\right)\right]\right) \] An empirical investigation of these bounds in the logistic classification model reveals that these approximations are great surrogates for the KL divergence. This result, and future results of a similar nature, could provide a path towards rigorously controlling the error due to the Laplace approximation and more modern approximation methods.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Sequential Gibbs Posteriors with Applications to Principal Component Analysis
Gibbs posteriors are proportional to a prior distribution multiplied by an exponentiated loss function, with a key tuning parameter weighting information in the loss relative to the prior and providing a control of poste…
Bayesian InferenceUncertainty QuantificationBernstein-von Mises for Adaptively Collected Data
Uncertainty quantification (UQ) for adaptively collected data, such as that coming from adaptive experiments, bandits, or reinforcement learning, is necessary for critical elements of data collection such as ensuring saf…
Reinforcement LearningVariable Selection with Rigorous Uncertainty Quantification using Deep Bayesian Neural Networks: Posterior Concentration and Bernstein-von Mises Phenomenon
This work develops rigorous theoretical basis for the fact that deep Bayesian neural network (BNN) is an effective tool for high-dimensional variable selection with rigorous uncertainty quantification. We develop new Bay…
Uncertainty QuantificationVariable SelectionOn Gaussian Process Priors in Conditional Moment Restriction Models
This paper studies quasi Bayesian estimation and uncertainty quantification for an unknown function that is identified by a nonparametric conditional moment restriction. We derive contraction rates for a class of Gaussia…
Uncertainty QuantificationDebiased Bayesian Inference for High-dimensional Regression Models
There has been significant progress in Bayesian inference based on sparsity-inducing (e.g., spike-and-slab and horseshoe-type) priors for high-dimensional regression models. The resulting posteriors, however, in general …
Bayesian Inference