Momentum-Based Policy Gradient Methods

2020-07-13 · ICML 2020 1 · Feihu Huang, Shangqian Gao, Jian Pei, Heng Huang

In the paper, we propose a class of efficient momentum-based policy gradient methods for the model-free reinforcement learning, which use adaptive learning rates and do not require any large batches. Specifically, we propose a fast important-sampling momentum-based policy gradient (IS-MBPG) method based on a new momentum-based variance reduced technique and the importance sampling technique. We also propose a fast Hessian-aided momentum-based policy gradient (HA-MBPG) method based on the momentum-based variance reduced technique and the Hessian-aided technique. Moreover, we prove that both the IS-MBPG and HA-MBPG methods reach the best known sample complexity of $O(\epsilon^{-3})$ for finding an $\epsilon$-stationary point of the non-concave performance function, which only require one trajectory at each iteration. In particular, we present a non-adaptive version of IS-MBPG method, i.e., IS-MBPG*, which also reaches the best known sample complexity of $O(\epsilon^{-3})$ without any large batches. In the experiments, we apply four benchmark tasks to demonstrate the effectiveness of our algorithms.

📄 PDF Abstract BibTeX arXiv:2007.06680

Code (1)

gaosh/MBPG 공식 구현 pytorch

Tasks

Policy Gradient Methods

Stochastic Recursive Momentum for Policy Gradient Methods

2020-03-09 · Huizhuo Yuan, Xiangru Lian, Ji Liu, Yuren Zhou

In this paper, we propose a novel algorithm named STOchastic Recursive Momentum for Policy Gradient (STORM-PG), which operates a SARAH-type stochastic recursive variance-reduced policy gradient in an exponential moving a…

Policy Gradient Methods

Global Convergence of Natural Policy Gradient with Hessian-aided Momentum Variance Reduction

2024-01-02 · Jie Feng, Ke Wei, Jinchi Chen

Natural policy gradient (NPG) and its variants are widely-used policy search methods in reinforcement learning. Inspired by prior work, a new NPG variant coined NPG-HM is developed in this paper, which utilizes the Hessi…

MuJoCo Policy Gradient Methods

MDPGT: Momentum-based Decentralized Policy Gradient Tracking

2021-12-06 · Zhanhong Jiang, Xian Yeow Lee, Sin Yong Tan, Kai Liang Tan 외

We propose a novel policy gradient method for multi-agent reinforcement learning, which leverages two different variance-reduction techniques and does not require large batches over iterations. Specifically, we propose a…

Multi-agent Reinforcement Learning Policy Gradient Methods reinforcement-learning Reinforcement Learning+2

On the Global Optimum Convergence of Momentum-based Policy Gradient

2021-10-19 · Yuhao Ding, Junzi Zhang, Javad Lavaei

Policy gradient (PG) methods are popular and efficient for large-scale reinforcement learning due to their relative stability and incremental nature. In recent years, the empirical success of PG methods has led to the de…

Correcting Momentum in Temporal Difference Learning

2021-06-07 · Emmanuel Bengio, Joelle Pineau, Doina Precup

A common optimization tool used in deep reinforcement learning is momentum, which consists in accumulating and discounting past gradients, reapplying them at each iteration. We argue that, unlike in supervised learning, …

Deep Reinforcement Learning Reinforcement Learning (RL)