A unified view of likelihood ratio and reparameterization gradients

2021-05-31 · Paavo Parmas, Masashi Sugiyama

Reparameterization (RP) and likelihood ratio (LR) gradient estimators are used to estimate gradients of expectations throughout machine learning and reinforcement learning; however, they are usually explained as simple mathematical tricks, with no insight into their nature. We use a first principles approach to explain that LR and RP are alternative methods of keeping track of the movement of probability mass, and the two are connected via the divergence theorem. Moreover, we show that the space of all possible estimators combining LR and RP can be completely parameterized by a flow field $u(x)$ and an importance sampling distribution $q(x)$. We prove that there cannot exist a single-sample estimator of this type outside our characterized space, thus, clarifying where we should be searching for better Monte Carlo gradient estimators.

📄 PDF Abstract BibTeX arXiv:2105.14900

Code (0)

등록된 구현이 없습니다.

A unified view of likelihood ratio and reparameterization gradients and an optimal importance sampling scheme

2019-10-14 · Paavo Parmas, Masashi Sugiyama

Reparameterization (RP) and likelihood ratio (LR) gradient estimators are used throughout machine and reinforcement learning; however, they are usually explained as simple mathematical tricks without providing any insigh…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Perturbative Black Box Variational Inference

2017-09-21 · NeurIPS 2017 12 · Robert Bamler, Cheng Zhang, Manfred Opper, Stephan Mandt

Black box variational inference (BBVI) with reparameterization gradients triggered the exploration of divergence measures other than the Kullback-Leibler (KL) divergence, such as alpha divergences. In this paper, we view…

Gaussian Processes Variational Inference

Evaluating the Variance of Likelihood-Ratio Gradient Estimators

2017-08-01 · ICML 2017 8 · Seiya Tokui, Issei Sato

The likelihood-ratio method is often used to estimate gradients of stochastic computations, for which baselines are required to reduce the estimation variance. Many types of baselines have been proposed, although th…

PIPPS: Flexible Model-Based Policy Search Robust to the Curse of Chaos

2019-02-04 · ICML 2018 7 · Paavo Parmas, Carl Edward Rasmussen, Jan Peters, Kenji Doya

Previously, the exploding gradient problem has been explained to be central in deep learning and model-based reinforcement learning, because it causes numerical issues and instability in optimization. Our experiments in …

Model-based Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Reparameterization Flow Policy Optimization

2026-02-03 · Hai Zhong, Zhuoran Li, Xun Wang, Longbo Huang arxiv

Reparameterization Policy Gradient (RPG) has emerged as a powerful paradigm for model-based reinforcement learning, enabling high sample efficiency by backpropagating gradients through differentiable dynamics. However, p…

Reinforcement Learning