Understanding the Variance Collapse of SVGD in High Dimensions
Stein variational gradient descent (SVGD) is a deterministic inference algorithm that evolves a set of particles to fit a target distribution. Despite its computational efficiency, SVGD often underestimates the variance of the target distribution in high dimensions. In this work we attempt to explain the variance collapse in SVGD. On the qualitative side, we compare the SVGD update with gradient descent on the maximum mean discrepancy (MMD) objective; we observe that the variance collapse phenomenon relates to the bias from deterministic updates present in the "driving force" of SVGD, and empirically verify that removal of such bias leads to more accurate variance estimation. On the quantitative side, we demonstrate that the variance collapse of SVGD can be accurately predicted in the proportional asymptotic limit, i.e., when the number of particles $n$ and dimensions $d$ diverge at the same rate. In particular, for learning high-dimensional isotropic Gaussians, we derive the exact equilibrium variance for both SVGD and MMD-descent, under certain near-orthogonality assumption on the converged particles, and confirm that SVGD suffers from the "curse of dimensionality".
Code (0)
등록된 구현이 없습니다.
Tasks
Computational EfficiencyVocal Bursts Intensity PredictionSimilar Papers 제목 키워드 기반
Variational Inference via Entropic Transport Descent
Particle-based variational inference (ParVI) methods approximate an intractable target distribution by evolving an ensemble of interacting samples. Existing approaches rely predominantly on kernel-based repulsion (e.g., …
Augmented Message Passing Stein Variational Gradient Descent
Stein Variational Gradient Descent (SVGD) is a popular particle-based method for Bayesian inference. However, its convergence suffers from the variance collapse, which reduces the accuracy and diversity of the estimation…
Bayesian InferenceDiversityMessage Passing Stein Variational Gradient Descent
Stein variational gradient descent (SVGD) is a recently proposed particle-based Bayesian inference method, which has attracted a lot of interest due to its remarkable approximation ability and particle efficiency compare…
Bayesian InferenceVariational InferenceELBOing Stein: Variational Bayes with Stein Mixture Inference
Stein variational gradient descent (SVGD) [Liu and Wang, 2016] performs approximate Bayesian inference by representing the posterior with a set of particles. However, SVGD suffers from variance collapse, i.e. poor predic…
Bayesian InferenceTowards Characterizing the High-dimensional Bias of Kernel-based Particle Inference Algorithms
Particle-based inference algorithm is a promising method to efficiently generate samples for an intractable target distribution by iteratively updating a set of particles. As a noticeable example, Stein variational gradi…
LEMMA