Sample Efficient Policy Gradient Methods with Recursive Variance Reduction

2019-09-18 · ICLR 2020 1 · Pan Xu, Felicia Gao, Quanquan Gu

Improving the sample efficiency in reinforcement learning has been a long-standing research problem. In this work, we aim to reduce the sample complexity of existing policy gradient methods. We propose a novel policy gradient algorithm called SRVR-PG, which only requires $O(1/\epsilon^{3/2})$ episodes to find an $\epsilon$-approximate stationary point of the nonconcave performance function $J(\boldsymbol{\theta})$ (i.e., $\boldsymbol{\theta}$ such that $\|\nabla J(\boldsymbol{\theta})\|_2^2\leq\epsilon$). This sample complexity improves the existing result $O(1/\epsilon^{5/3})$ for stochastic variance reduced policy gradient algorithms by a factor of $O(1/\epsilon^{1/6})$. In addition, we also propose a variant of SRVR-PG with parameter exploration, which explores the initial policy parameter from a prior probability distribution. We conduct numerical experiments on classic control problems in reinforcement learning to validate the performance of our proposed algorithms.

📄 PDF Abstract BibTeX arXiv:1909.08610

Code (1)

xgfelicia/SRVRPG pytorch

Tasks

Policy Gradient Methodsreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Stochastic Recursive Momentum for Policy Gradient Methods

2020-03-09 · Huizhuo Yuan, Xiangru Lian, Ji Liu, Yuren Zhou

In this paper, we propose a novel algorithm named STOchastic Recursive Momentum for Policy Gradient (STORM-PG), which operates a SARAH-type stochastic recursive variance-reduced policy gradient in an exponential moving a…

Policy Gradient Methods

Policy Optimization via Stochastic Recursive Gradient Algorithm

2019-05-01 · ICLR 2019 5 · Huizhuo Yuan, Chris Junchi Li, Yuhao Tang, Yuren Zhou

In this paper, we propose the StochAstic Recursive grAdient Policy Optimization (SARAPO) algorithm which is a novel variance reduction method on Trust Region Policy Optimization (TRPO). The algorithm incorporates the Sto…

Trajectory-wise Control Variates for Variance Reduction in Policy Gradient Methods

2019-08-08 · Ching-An Cheng, Xinyan Yan, Byron Boots

Policy gradient methods have demonstrated success in reinforcement learning tasks that have high-dimensional continuous state and action spaces. However, policy gradient methods are also notoriously sample inefficient. T…

Policy Gradient Methods Reinforcement Learning

A Temporal-Difference Approach to Policy Gradient Estimation

2022-02-04 · Samuele Tosatto, Andrew Patterson, Martha White, A. Rupam Mahmood

The policy gradient theorem (Sutton et al., 2000) prescribes the usage of a cumulative discounted state distribution under the target policy to approximate the gradient. Most algorithms based on this theorem, in practice…

PAGE-PG: A Simple and Loopless Variance-Reduced Policy Gradient Method with Probabilistic Gradient Estimation

2022-02-01 · Matilde Gargiani, Andrea Zanelli, Andrea Martinelli, Tyler Summers 외

Despite their success, policy gradient methods suffer from high variance of the gradient estimate, which can result in unsatisfactory sample complexity. Recently, numerous variance-reduced extensions of policy gradient m…