Understanding Variational Inference in Function-Space
Recent work has attempted to directly approximate the function-space' or predictive posterior distribution of Bayesian models, without approximating the posterior distribution over the parameters. This is appealing in e.g. Bayesian neural networks, where we only need the former, and the latter is hard to represent. In this work, we highlight some advantages and limitations of employing the Kullback-Leibler divergence in this setting. For example, we show that minimizing the KL divergence between a wide class of parametric distributions and the posterior induced by a (non-degenerate) Gaussian process prior leads to an ill-defined objective function. Then, we propose (featurized) Bayesian linear regression as a benchmark for function-space' inference methods that directly measures approximation quality. We apply this methodology to assess aspects of the objective function and inference scheme considered in Sun, Zhang, Shi, and Grosse (2018), emphasizing the quality of approximation to Bayesian inference as opposed to predictive performance.
Code (2)
Tasks
Bayesian InferenceVariational InferenceMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Rethinking Function-Space Variational Inference in Bayesian Neural Networks
Bayesian neural networks (BNNs) define distributions over functions induced by distributions over parameters. In practice, this model specification makes it difficult to define and use meaningful prior distributions over…
Variational InferenceGeneralized Variational Inference in Function Spaces: Gaussian Measures meet Bayesian Deep Learning
We develop a framework for generalized variational inference in infinite-dimensional function spaces and use it to construct a method termed Gaussian Wasserstein inference (GWI). GWI leverages the Wasserstein distance be…
Gaussian ProcessesUncertainty QuantificationVariational InferenceFunction-Space Variational Inference for Deep Bayesian Classification
Bayesian deep learning approaches assume model parameters to be latent random variables and infer posterior predictive distributions to quantify uncertainty, increase safety and trust, and prevent overconfident and unpre…
Adversarial RobustnessClassificationimage-classificationImage Classification+2Functional Variational Inference based on Stochastic Process Generators
Bayesian inference in the space of functions has been an important topic for Bayesian modeling in the past. In this paper, we propose a new solution to this problem called Functional Variational Inference (FVI). In FVI, …
Bayesian InferenceVariational InferenceFunctional Bias and Tangent-Space Geometry in Variational Inference
Variational inference approximates Bayesian posterior distributions by projecting onto a tractable family of distributions. While most theoretical analyses evaluate the quality of this approximation using global divergen…