REBAR: Low-variance, unbiased gradient estimates for discrete latent variable models
Learning in models with discrete latent variables is challenging due to high variance gradient estimators. Generally, approaches have relied on control variates to reduce the variance of the REINFORCE estimator. Recent work (Jang et al. 2016, Maddison et al. 2016) has taken a different approach, introducing a continuous relaxation of discrete variables to produce low-variance, but biased, gradient estimates. In this work, we combine the two approaches through a novel control variate that produces low-variance, \emph{unbiased} gradient estimates. Then, we introduce a modification to the continuous relaxation and show that the tightness of the relaxation can be adapted online, removing it as a hyperparameter. We show state-of-the-art variance reduction on several benchmark generative modeling tasks, generally leading to faster convergence to a better final log-likelihood.
Code (3)
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Backpropagation through the Void: Optimizing control variates for black-box gradient estimation
Gradient-based optimization is the foundation of deep learning and reinforcement learning. Even when the mechanism being optimized is unknown or not differentiable, optimization using high-variance or biased gradient est…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Adversarial Training of Neural Encoding Models on Population Spike Trains
Neural population responses to sensory stimuli can exhibit both nonlinear stimulus- dependence and richly structured shared variability. Here, we show how adversarial training can be used to optimize neural encoding mode…
Biased Gradient Estimate with Drastic Variance Reduction for Meta Reinforcement Learning
Despite the empirical success of meta reinforcement learning (meta-RL), there are still a number poorly-understood discrepancies between theory and practice. Critically, biased gradient estimates are almost always implem…
Meta Reinforcement Learningreinforcement-learningReinforcement Learning (RL)An Empirical Analysis of Measure-Valued Derivatives for Policy Gradients
Reinforcement learning methods for robotics are increasingly successful due to the constant development of better policy gradient techniques. A precise (low variance) and accurate (low bias) gradient estimator is crucial…
Low Bias Low Variance Gradient Estimates for Hierarchical Boolean Stochastic Networks
Stochastic neural networks with discrete random variables are an important class of models for their expressiveness and interpretability. Since direct differentiation and backpropagation is not possible, Monte Carlo grad…