On-Policy Policy Gradient Reinforcement Learning Without On-Policy Sampling

2023-11-14 · Nicholas E. Corrado, Josiah P. Hanna

On-policy reinforcement learning (RL) algorithms perform policy updates using i.i.d. trajectories collected by the current policy. However, after observing only a finite number of trajectories, on-policy sampling may produce data that fails to match the expected on-policy data distribution. This sampling error leads to noisy updates and data inefficient on-policy learning. Recent work in the policy evaluation setting has shown that non-i.i.d., off-policy sampling can produce data with lower sampling error than on-policy sampling can produce (Zhong et. al, 2022). Motivated by this observation, we introduce an adaptive, off-policy sampling method to improve the data efficiency of on-policy policy gradient algorithms. Our method, Proximal Robust On-Policy Sampling (PROPS), reduces sampling error by collecting data with a behavior policy that increases the probability of sampling actions that are under-sampled with respect to the current policy. We empirically evaluate PROPS on both continuous-action MuJoCo benchmark tasks as well discrete-action tasks and demonstrate that (1) PROPS decreases sampling error throughout training and (2) improves the data efficiency of on-policy policy gradient algorithms.

📄 PDF Abstract BibTeX arXiv:2311.08290

Code (0)

등록된 구현이 없습니다.

Tasks

MuJoCoreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Batch Reinforcement Learning with Hyperparameter Gradients

2020-01-01 · ICML 2020 1 · Byung-Jun Lee, Jongmin Lee, Peter Vrancx, Dongho Kim 외

We consider the batch reinforcement learning problem where the agent needs to learn only from a fixed batch of data, without further interaction with the environment. In such a scenario, we want to prevent the optimized …

continuous-control Continuous Control reinforcement-learning Reinforcement Learning+1

Self-Supervised Continuous Control without Policy Gradient

2021-01-01 · Hao Sun, Ziping Xu, Meng Fang, Yuhang Song 외

Despite the remarkable progress made by the policy gradient algorithms in reinforcement learning (RL), sub-optimal policies usually result from the local exploration property of the policy gradient update. In this work, …

continuous-control Continuous Control MuJoCo Policy Gradient Methods+3

Statistically Efficient Off-Policy Policy Gradients

2020-02-10 · ICML 2020 1 · Nathan Kallus, Masatoshi Uehara

Policy gradient methods in reinforcement learning update policy parameters by taking steps in the direction of an estimated gradient of policy value. In this paper, we consider the statistically efficient estimation of p…

Policy Gradient Methods Reinforcement Learning

Dynamical System Optimization

2025-06-10 · Emo Todorov

We develop an optimization framework centered around a core idea: once a (parametric) policy is specified, control authority is transferred to the policy, resulting in an autonomous dynamical system. Thus we should be ab…

reinforcement-learning Reinforcement Learning

Equivalence Between Policy Gradients and Soft Q-Learning

2017-04-21 · John Schulman, Xi Chen, Pieter Abbeel

Two of the leading approaches for model-free reinforcement learning are policy gradient methods and $Q$-learning methods. $Q$-learning methods can be effective and sample-efficient when they work, however, it is not well…

Policy Gradient Methods Q-Learning reinforcement-learning Reinforcement Learning+1