Efficient Evaluation of Natural Stochastic Policies in Offline Reinforcement Learning

2020-06-06 · Nathan Kallus, Masatoshi Uehara

We study the efficient off-policy evaluation of natural stochastic policies, which are defined in terms of deviations from the behavior policy. This is a departure from the literature on off-policy evaluation where most work consider the evaluation of explicitly specified policies. Crucially, offline reinforcement learning with natural stochastic policies can help alleviate issues of weak overlap, lead to policies that build upon current practice, and improve policies' implementability in practice. Compared with the classic case of a pre-specified evaluation policy, when evaluating natural stochastic policies, the efficiency bound, which measures the best-achievable estimation error, is inflated since the evaluation policy itself is unknown. In this paper, we derive the efficiency bounds of two major types of natural stochastic policies: tilting policies and modified treatment policies. We then propose efficient nonparametric estimators that attain the efficiency bounds under very lax conditions. These also enjoy a (partial) double robustness property.

📄 PDF Abstract BibTeX arXiv:2006.03886

Code (0)

등록된 구현이 없습니다.

Tasks

Off-policy evaluationreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Uncertainty-aware Distributional Offline Reinforcement Learning

2024-03-26 · Xiaocong Chen, Siyu Wang, Tong Yu, Lina Yao

Offline reinforcement learning (RL) presents distinct challenges as it relies solely on observational data. A central concern in this context is ensuring the safety of the learned policy by quantifying uncertainties asso…

Offline RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Operator Models for Continuous-Time Offline Reinforcement Learning

2025-11-13 · Nicolas Hoischen, Petar Bevanda, Max Beier, Stefan Sosnowski 외 arxiv

Continuous-time stochastic processes underlie many natural and engineered systems. In healthcare, autonomous driving, and industrial control, direct interaction with the environment is often unsafe or impractical, motiva…

Reinforcement Learning Autonomous Driving

Risk-Averse Offline Reinforcement Learning

2021-02-10 · ICLR 2021 1 · Núria Armengol Urpí, Sebastian Curi, Andreas Krause

Training Reinforcement Learning (RL) agents in high-stakes applications might be too prohibitive due to the risk associated to exploration. Thus, the agent can only use data previously collected by safe policies. While p…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Streetwise Agents: Empowering Offline RL Policies to Outsmart Exogenous Stochastic Disturbances in RTC

2024-11-11 · Aditya Soni, Mayukh Das, Anjaly Parayil, Supriyo Ghosh 외

The difficulty of exploring and training online on real production systems limits the scope of real-time online data/feedback-driven decision making. The most feasible approach is to adopt offline reinforcement learning …

Offline RL

Representation Balancing Offline Model-based Reinforcement Learning

2021-01-01 · ICLR 2021 1 · Byung-Jun Lee, Jongmin Lee, Kee-Eung Kim

One of the main challenges in offline and off-policy reinforcement learning is to cope with the distribution shift that arises from the mismatch between the target policy and the data collection policy. In this paper, we…

model Model-based Reinforcement Learning Offline RL reinforcement-learning+2