Doubly Robust Off-Policy Value and Gradient Estimation for Deterministic Policies

2020-06-06 · NeurIPS 2020 12 · Nathan Kallus, Masatoshi Uehara

Offline reinforcement learning, wherein one uses off-policy data logged by a fixed behavior policy to evaluate and learn new policies, is crucial in applications where experimentation is limited such as medicine. We study the estimation of policy value and gradient of a deterministic policy from off-policy data when actions are continuous. Targeting deterministic policies, for which action is a deterministic function of state, is crucial since optimal policies are always deterministic (up to ties). In this setting, standard importance sampling and doubly robust estimators for policy value and gradient fail because the density ratio does not exist. To circumvent this issue, we propose several new doubly robust estimators based on different kernelization approaches. We analyze the asymptotic mean-squared error of each of these under mild rate conditions for nuisance estimators. Specifically, we demonstrate how to obtain a rate that is independent of the horizon length.

📄 PDF Abstract BibTeX arXiv:2006.03900

Code (1)

CausalML/DoublyRobustOPEForDeterministicPolicies

Doubly Robust Off-Policy Actor-Critic Algorithms for Reinforcement Learning

2019-12-11 · Riashat Islam, Raihan Seraj, Samin Yeasar Arnob, Doina Precup

We study the problem of off-policy critic evaluation in several variants of value-based off-policy actor-critic algorithms. Off-policy actor-critic algorithms require an off-policy critic evaluation step, to estimate the…

continuous-control Continuous Control reinforcement-learning Reinforcement Learning+2

Softmax Deep Double Deterministic Policy Gradients

2020-10-19 · NeurIPS 2020 12 · Ling Pan, Qingpeng Cai, Longbo Huang

A widely-used actor-critic reinforcement learning algorithm for continuous control, Deep Deterministic Policy Gradients (DDPG), suffers from the overestimation problem, which can negatively affect the performance. Althou…

continuous-control Continuous Control

Bayesian Policy Gradients via Alpha Divergence Dropout Inference

2017-12-06 · Peter Henderson, Thang Doan, Riashat Islam, David Meger

Policy gradient methods have had great success in solving continuous control tasks, yet the stochastic nature of such problems makes deterministic value estimation difficult. We propose an approach which instead estimate…

continuous-control Continuous Control MuJoCo Policy Gradient Methods

Doubly Robust Policy Evaluation and Optimization

2015-03-10 · Miroslav Dudík, Dumitru Erhan, John Langford, Lihong Li

We study sequential decision making in environments where rewards are only partially observed, but can be modeled as a function of observed contexts and the chosen action by the decision maker. This setting, known as con…

Decision Making Multi-Armed Bandits Sequential Decision Making

Deterministic Value-Policy Gradients

2019-09-09 · Qingpeng Cai, Ling Pan, Pingzhong Tang

Reinforcement learning algorithms such as the deep deterministic policy gradient algorithm (DDPG) has been widely used in continuous control tasks. However, the model-free DDPG algorithm suffers from high sample complexi…

continuous-control Continuous Control Deep Reinforcement Learning reinforcement-learning+2