Differentiable Antithetic Sampling for Variance Reduction in Stochastic Variational Inference
Stochastic optimization techniques are standard in variational inference algorithms. These methods estimate gradients by approximating expectations with independent Monte Carlo samples. In this paper, we explore a technique that uses correlated, but more representative , samples to reduce estimator variance. Specifically, we show how to generate antithetic samples that match sample moments with the true moments of an underlying importance distribution. Combining a differentiable antithetic sampler with modern stochastic variational inference, we showcase the effectiveness of this approach for learning a deep generative model.
Code (1)
Tasks
Stochastic OptimizationVariational InferenceSimilar Papers 제목 키워드 기반
Accelerating Stochastic Gradient Descent Using Antithetic Sampling
(Mini-batch) Stochastic Gradient Descent is a popular optimization method which has been applied to many machine learning applications. But a rather high variance introduced by the stochastic gradient in each step may sl…
Binary ClassificationGeneral ClassificationStochastic Variational Optimization
Variational Optimization forms a differentiable upper bound on an objective. We show that approaches such as Natural Evolution Strategies and Gaussian Perturbation, are special cases of Variational Optimization in which …
ARM: Augment-REINFORCE-Merge Gradient for Stochastic Binary Networks
To backpropagate the gradients through stochastic binary layers, we propose the augment-REINFORCE-merge (ARM) estimator that is unbiased, exhibits low variance, and has low computational complexity. Exploiting variable a…
Data AugmentationVariational InferenceDisARM: An Antithetic Gradient Estimator for Binary Latent Variables
Training models with discrete latent variables is challenging due to the difficulty of estimating the gradients accurately. Much of the recent progress has been achieved by taking advantage of continuous relaxations of t…
Uncertainty-Aware Distribution-to-Distribution Flow Matching for Scientific Imaging
Distribution-to-distribution generative models support scientific imaging tasks ranging from modeling cellular perturbation responses to translating medical images across conditions. Trustworthy generation requires relia…