Variational Inference: A Review for Statisticians
One of the core problems of modern statistics is to approximate difficult-to-compute probability densities. This problem is especially important in Bayesian statistics, which frames all inference about unknown quantities as a calculation involving the posterior density. In this paper, we review variational inference (VI), a method from machine learning that approximates probability densities through optimization. VI has been used in many applications and tends to be faster than classical methods, such as Markov chain Monte Carlo sampling. The idea behind VI is to first posit a family of densities and then to find the member of that family which is close to the target. Closeness is measured by Kullback-Leibler divergence. We review the ideas behind mean-field variational inference, discuss the special case of VI applied to exponential family models, present a full example with a Bayesian mixture of Gaussians, and derive a variant that uses stochastic optimization to scale up to massive data. We discuss modern research in VI and highlight important open problems. VI is powerful, but it is not yet well understood. Our hope in writing this paper is to catalyze statistical research on this class of algorithms.
Code (6)
Tasks
Stochastic OptimizationVariational InferenceSimilar Papers 제목 키워드 기반
Scalable GAM using sparse variational Gaussian processes
Generalized additive models (GAMs) are a widely used class of models of interest to statisticians as they provide a flexible way to design interpretable models of data beyond linear models. We here propose a scalable and…
Additive modelsGaussian ProcessesVariational InferenceA Frequentist Statistical Introduction to Variational Inference, Autoencoders, and Diffusion Models
While Variational Inference (VI) is central to modern generative models like Variational Autoencoders (VAEs) and Denoising Diffusion Models (DDMs), its pedagogical treatment is split across disciplines. In statistics, VI…
An Introduction to Quantum Computing for Statisticians and Data Scientists
Quantum computers promise to surpass the most powerful classical supercomputers when it comes to solving many critically important practical problems, such as pharmaceutical and fertilizer design, supply chain and traffi…
On The Problem of Relevance in Statistical Inference
This paper is dedicated to the "50 Years of the Relevance Problem" - a long-neglected topic that begs attention from practical statisticians who are concerned with the problem of drawing inference from large-scale hetero…
Advances in Variational Inference
Many modern unsupervised or semi-supervised machine learning algorithms rely on Bayesian probabilistic models. These models are usually intractable and thus require approximate inference. Variational inference (VI) lets …
Variational Inference