Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processes

2019-08-22 · Nathan Kallus, Masatoshi Uehara

Off-policy evaluation (OPE) in reinforcement learning allows one to evaluate novel decision policies without needing to conduct exploration, which is often costly or otherwise infeasible. We consider for the first time the semiparametric efficiency limits of OPE in Markov decision processes (MDPs), where actions, rewards, and states are memoryless. We show existing OPE estimators may fail to be efficient in this setting. We develop a new estimator based on cross-fold estimation of $q$-functions and marginalized density ratios, which we term double reinforcement learning (DRL). We show that DRL is efficient when both components are estimated at fourth-root rates and is also doubly robust when only one component is consistent. We investigate these properties empirically and demonstrate the performance benefits due to harnessing memorylessness.

📄 PDF Abstract BibTeX arXiv:1908.08526

Code (1)

CausalML/DoubleReinforcementLearningMDP 공식 구현 pytorch

Tasks

Off-policy evaluationreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Double Reinforcement Learning for Efficient and Robust Off-Policy Evaluation

2020-01-01 · ICML 2020 1 · Nathan Kallus, Masatoshi Uehara

Off-policy evaluation reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Efficiently Breaking the Curse of Horizon in Off-Policy Evaluation with Double Reinforcement Learning

2019-09-12 · Nathan Kallus, Masatoshi Uehara

Off-policy evaluation (OPE) in reinforcement learning is notoriously difficult in long- and infinite-horizon settings due to diminishing overlap between behavior and target policies. In this paper, we study the role of M…

Off-policy evaluation reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Reinforcement Learning: Tutorial and Survey

2024-07-18 · OSF Preprints 2024 7 · Benyamin Ghojogh, Ali Ghodsi

This is a tutorial and survey paper on reinforcement learning, from fundamental reinforcement learning to deep reinforcement learning. It starts with introducing the elements of reinforcement learning. Then, Markov decis…

Deep Reinforcement Learning General Reinforcement Learning Q-Learning reinforcement-learning+3

Adaptive Double-Booking Strategy for Outpatient Scheduling Using Multi-Objective Reinforcement Learning

2026-03-07 · Ninda Nurseha Amalina, Heungjo An arxiv

Patient no-shows disrupt outpatient clinic operations, reduce productivity, and may delay necessary care. Clinics often adopt overbooking or double-booking to mitigate these effects. However, poorly calibrated policies c…

Reinforcement Learning

Statistically Efficient Off-Policy Policy Gradients

2020-02-10 · ICML 2020 1 · Nathan Kallus, Masatoshi Uehara

Policy gradient methods in reinforcement learning update policy parameters by taking steps in the direction of an estimated gradient of policy value. In this paper, we consider the statistically efficient estimation of p…

Policy Gradient Methods Reinforcement Learning