Off-policy Evaluation in Infinite-Horizon Reinforcement Learning with Latent Confounders

2020-07-27 · Andrew Bennett, Nathan Kallus, Lihong Li, Ali Mousavi

Off-policy evaluation (OPE) in reinforcement learning is an important problem in settings where experimentation is limited, such as education and healthcare. But, in these very same settings, observed actions are often confounded by unobserved variables making OPE even more difficult. We study an OPE problem in an infinite-horizon, ergodic Markov decision process with unobserved confounders, where states and actions can act as proxies for the unobserved confounders. We show how, given only a latent variable model for states and actions, policy value can be identified from off-policy data. Our method involves two stages. In the first, we show how to use proxies to estimate stationary distribution ratios, extending recent work on breaking the curse of horizon to the confounded setting. In the second, we show optimal balancing can be combined with such learned ratios to obtain policy value while avoiding direct modeling of reward functions. We establish theoretical guarantees of consistency, and benchmark our method empirically.

📄 PDF Abstract BibTeX arXiv:2007.13893

Code (0)

등록된 구현이 없습니다.

Tasks

Off-policy evaluationreinforcement-learning Reinforcement Learning (RL)

Confounding-Robust Policy Evaluation in Infinite-Horizon Reinforcement Learning

2020-02-11 · NeurIPS 2020 12 · Nathan Kallus, Angela Zhou

Off-policy evaluation of sequential decision policies from observational data is necessary in applications of batch reinforcement learning such as education and healthcare. In such settings, however, unobserved variables…

Off-policy evaluation reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1

On the Sample Complexity of Vanilla Model-Based Offline Reinforcement Learning with Dependent Samples

2023-03-07 · Mustafa O. Karabag, Ufuk Topcu

Offline reinforcement learning (offline RL) considers problems where learning is performed using only previously collected samples and is helpful for the settings in which collecting new data is costly or risky. In model…

Offline RL Off-policy evaluation reinforcement-learning Reinforcement Learning+1

Beyond dynamic programming

2023-06-26 · Abhinav Muraleedharan

In this paper, we present Score-life programming, a novel theoretical approach for solving reinforcement learning problems. In contrast with classical dynamic programming-based methods, our method can search over non-sta…

reinforcement-learning Reinforcement Learning

Infinite Time Horizon Safety of Bayesian Neural Networks

2021-11-04 · NeurIPS 2021 12 · Mathias Lechner, Đorđe Žikelić, Krishnendu Chatterjee, Thomas A. Henzinger

Bayesian neural networks (BNNs) place distributions over the weights of a neural network to model uncertainty in the data and the network's prediction. We consider the problem of verifying safety when running a Bayesian …

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Safe Exploration

Statistical Inference of the Value Function for Reinforcement Learning in Infinite Horizon Settings

2020-01-13 · C. Shi, S. Zhang, W. Lu, R. Song

Reinforcement learning is a general technique that allows an agent to learn an optimal policy and interact with an environment in sequential decision making problems. The goodness of a policy is measured by its value fun…

Decision Making reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1