Analyzing the Variance of Policy Gradient Estimators for the Linear-Quadratic Regulator

2019-10-02 · James A. Preiss, Sébastien M. R. Arnold, Chen-Yu Wei, Marius Kloft

We study the variance of the REINFORCE policy gradient estimator in environments with continuous state and action spaces, linear dynamics, quadratic cost, and Gaussian noise. These simple environments allow us to derive bounds on the estimator variance in terms of the environment and noise parameters. We compare the predictions of our bounds to the empirical variance in simulation experiments.

📄 PDF Abstract BibTeX arXiv:1910.01249

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

REINFORCE REINFORCE is a Monte Carlo variant of a policy gradient algorithm in reinforcement learning. The agent collects samples of an episode using its current policy, and uses it to…

Augment-Reinforce-Merge Policy Gradient for Binary Stochastic Policy

2019-03-13 · Yunhao Tang, Mingzhang Yin, Mingyuan Zhou

Due to the high variance of policy gradients, on-policy optimization algorithms are plagued with low sample efficiency. In this work, we propose Augment-Reinforce-Merge (ARM) policy gradient estimator as an unbiased low-…

Marginal Policy Gradients: A Unified Family of Estimators for Bounded Action Spaces with Applications

2018-06-13 · ICLR 2019 5 · Carson Eisenach, Haichuan Yang, Ji Liu, Han Liu

Many complex domains, such as robotics control and real-time strategy (RTS) games, require an agent to learn a continuous control. In the former, an agent learns a policy over $\mathbb{R}^d$ and in the latter, over a dis…

continuous-control Continuous Control Reinforcement Learning Reinforcement Learning (RL)

From Importance Sampling to Doubly Robust Policy Gradient

2019-10-20 · ICML 2020 1 · Jiawei Huang, Nan Jiang

We show that on-policy policy gradient (PG) and its variance reduction variants can be derived by taking finite difference of function evaluations supplied by estimators from the importance sampling (IS) family for off-p…

Off-policy evaluation

Does "Do Differentiable Simulators Give Better Policy Gradients?'' Give Better Policy Gradients?

2026-04-20 · Ku Onoda, Paavo Parmas, Manato Yaguchi, Yutaka Matsuo arxiv

In policy gradient reinforcement learning, access to a differentiable model enables 1st-order gradient estimation that accelerates learning compared to relying solely on derivative-free 0th-order estimators. However, dis…

Reinforcement Learning

Variance-Reduced Off-Policy Memory-Efficient Policy Search

2020-09-14 · Daoming Lyu, Qi Qi, Mohammad Ghavamzadeh, Hengshuai Yao 외

Off-policy policy optimization is a challenging problem in reinforcement learning (RL). The algorithms designed for this problem often suffer from high variance in their estimators, which results in poor sample efficienc…

Reinforcement Learning (RL)Stochastic Optimization