Confounding-Robust Policy Evaluation in Infinite-Horizon Reinforcement Learning
Off-policy evaluation of sequential decision policies from observational data is necessary in applications of batch reinforcement learning such as education and healthcare. In such settings, however, unobserved variables confound observed actions, rendering exact evaluation of new policies impossible, i.e., unidentifiable. We develop a robust approach that estimates sharp bounds on the (unidentifiable) value of a given policy in an infinite-horizon problem given data from another policy with unobserved confounding, subject to a sensitivity model. We consider stationary or baseline unobserved confounding and compute bounds by optimizing over the set of all stationary state-occupancy ratios that agree with a new partially identified estimating equation and the sensitivity model. We prove convergence to the sharp bounds as we collect more confounded data. Although checking set membership is a linear program, the support function is given by a difficult nonconvex optimization problem. We develop approximations based on nonconvex projected gradient descent and demonstrate the resulting bounds empirically.
Code (0)
등록된 구현이 없습니다.
Tasks
Off-policy evaluationreinforcement-learningReinforcement LearningReinforcement Learning (RL)SensitivitySimilar Papers 제목 키워드 기반
Reinforcement Learning with Continuous Actions Under Unmeasured Confounding
This paper addresses the challenge of offline policy learning in reinforcement learning with continuous action spaces when unmeasured confounders are present. While most existing research focuses on policy evaluation wit…
reinforcement-learningReinforcement LearningOff-policy Evaluation in Infinite-Horizon Reinforcement Learning with Latent Confounders
Off-policy evaluation (OPE) in reinforcement learning is an important problem in settings where experimentation is limited, such as education and healthcare. But, in these very same settings, observed actions are often c…
Off-policy evaluationreinforcement-learningReinforcement Learning (RL)On the Sample Complexity of Vanilla Model-Based Offline Reinforcement Learning with Dependent Samples
Offline reinforcement learning (offline RL) considers problems where learning is performed using only previously collected samples and is helpful for the settings in which collecting new data is costly or risky. In model…
Offline RLOff-policy evaluationreinforcement-learningReinforcement Learning+1Beyond dynamic programming
In this paper, we present Score-life programming, a novel theoretical approach for solving reinforcement learning problems. In contrast with classical dynamic programming-based methods, our method can search over non-sta…
reinforcement-learningReinforcement LearningInfinite Time Horizon Safety of Bayesian Neural Networks
Bayesian neural networks (BNNs) place distributions over the weights of a neural network to model uncertainty in the data and the network's prediction. We consider the problem of verifying safety when running a Bayesian …
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Safe Exploration