Batch Reinforcement Learning with a Nonparametric Off-Policy Policy Gradient

2020-10-27 · Samuele Tosatto, João Carvalho, Jan Peters

Off-policy Reinforcement Learning (RL) holds the promise of better data efficiency as it allows sample reuse and potentially enables safe interaction with the environment. Current off-policy policy gradient methods either suffer from high bias or high variance, delivering often unreliable estimates. The price of inefficiency becomes evident in real-world scenarios such as interaction-driven robot learning, where the success of RL has been rather limited, and a very high sample cost hinders straightforward application. In this paper, we propose a nonparametric Bellman equation, which can be solved in closed form. The solution is differentiable w.r.t the policy parameters and gives access to an estimation of the policy gradient. In this way, we avoid the high variance of importance sampling approaches, and the high bias of semi-gradient methods. We empirically analyze the quality of our gradient estimate against state-of-the-art methods, and show that it outperforms the baselines in terms of sample efficiency on classical control tasks.

📄 PDF Abstract BibTeX arXiv:2010.14771

Code (0)

등록된 구현이 없습니다.

Tasks

Policy Gradient Methodsreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Batch Reinforcement Learning with Hyperparameter Gradients

2020-01-01 · ICML 2020 1 · Byung-Jun Lee, Jongmin Lee, Peter Vrancx, Dongho Kim 외

We consider the batch reinforcement learning problem where the agent needs to learn only from a fixed batch of data, without further interaction with the environment. In such a scenario, we want to prevent the optimized …

continuous-control Continuous Control reinforcement-learning Reinforcement Learning+1

A Nonparametric Off-Policy Policy Gradient

2020-01-08 · Samuele Tosatto, Joao Carvalho, Hany Abdulsamad, Jan Peters

Reinforcement learning (RL) algorithms still suffer from high sample complexity despite outstanding recent successes. The need for intensive interactions with the environment is especially observed in many widely popular…

Density Estimation Policy Gradient Methods Reinforcement Learning Reinforcement Learning (RL)

Batched Nonparametric Contextual Bandits

2024-02-27 · Rong Jiang, Cong Ma

We study nonparametric contextual bandits under batch constraints, where the expected reward for each action is modeled as a smooth function of covariates, and the policy updates are made at the end of each batch of obse…

Multi-Armed Bandits

Simultaneous Perturbation Algorithms for Batch Off-Policy Search

2014-03-18 · Raphael Fonteneau, L. A. Prashanth

We propose novel policy search algorithms in the context of off-policy, batch mode reinforcement learning (RL) with continuous state and action spaces. Given a batch collection of trajectories, we perform off-line policy…

Reinforcement Learning Reinforcement Learning (RL)

Adaptive Batch Size for Safe Policy Gradients

2017-12-01 · NeurIPS 2017 12 · Matteo Papini, Matteo Pirotta, Marcello Restelli

Policy gradient methods are among the best Reinforcement Learning (RL) techniques to solve complex control problems. In real-world RL applications, it is common to have a good initial policy whose performance needs to be…

Policy Gradient Methods Reinforcement Learning Reinforcement Learning (RL)