Robust On-Policy Sampling for Data-Efficient Policy Evaluation in Reinforcement Learning

2021-11-29 · Rujie Zhong, Duohan Zhang, Lukas Schäfer, Stefano V. Albrecht, Josiah P. Hanna

Reinforcement learning (RL) algorithms are often categorized as either on-policy or off-policy depending on whether they use data from a target policy of interest or from a different behavior policy. In this paper, we study a subtle distinction between on-policy data and on-policy sampling in the context of the RL sub-problem of policy evaluation. We observe that on-policy sampling may fail to match the expected distribution of on-policy data after observing only a finite number of trajectories and this failure hinders data-efficient policy evaluation. Towards improved data-efficiency, we show how non-i.i.d., off-policy sampling can produce data that more closely matches the expected on-policy data distribution and consequently increases the accuracy of the Monte Carlo estimator for policy evaluation. We introduce a method called Robust On-Policy Sampling and demonstrate theoretically and empirically that it produces data that converges faster to the expected on-policy distribution compared to on-policy sampling. Empirically, we show that this faster convergence leads to lower mean squared error policy value estimates.

📄 PDF Abstract BibTeX arXiv:2111.14552

Code (1)

uoe-agents/robust_onpolicy_data_collection 공식 구현 pytorch

Tasks

Offline RL reinforcement-learning Reinforcement Learning (RL)

On-Policy Policy Gradient Reinforcement Learning Without On-Policy Sampling

2023-11-14 · Nicholas E. Corrado, Josiah P. Hanna

On-policy reinforcement learning (RL) algorithms perform policy updates using i.i.d. trajectories collected by the current policy. However, after observing only a finite number of trajectories, on-policy sampling may pro…

MuJoCo reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Low Variance Off-policy Evaluation with State-based Importance Sampling

2022-12-07 · David M. Bossens, Philip S. Thomas

In many domains, the exploration process of reinforcement learning will be too costly as it requires trying out suboptimal policies, resulting in a need for off-policy evaluation, in which a target policy is evaluated ba…

Density Ratio Estimation Off-policy evaluation

Importance Resampling for Off-policy Policy Evaluation

2018-09-27 · Matthew Schlegel, Wesley Chung, Daniel Graves, Martha White

Importance sampling is a common approach to off-policy learning in reinforcement learning. While it is consistent and unbiased, it can result in high variance updates to the parameters for the value function. Weighted i…

The $f$-Divergence Reinforcement Learning Framework

2021-09-24 · Chen Gong, Qiang He, Yunpeng Bai, Zhou Yang 외

The framework of deep reinforcement learning (DRL) provides a powerful and widely applicable mathematical formalization for sequential decision-making. This paper present a novel DRL framework, termed \emph{$f$-Divergenc…

Decision Making Deep Reinforcement Learning Mathematical Proofs reinforcement-learning+3

Off-Policy Evaluation in Partially Observable Environments

2019-09-09 · Guy Tennenholtz, Shie Mannor, Uri Shalit

This work studies the problem of batch off-policy evaluation for Reinforcement Learning in partially observable environments. Off-policy evaluation under partial observability is inherently prone to bias, with risk of ar…

Off-policy evaluation Reinforcement Learning Reinforcement Learning (RL)