Statistical Estimation of Confounded Linear MDPs: An Instrumental Variable Approach

2022-09-12 · Miao Lu, Wenhao Yang, Liangyu Zhang, Zhihua Zhang

In an Markov decision process (MDP), unobservable confounders may exist and have impacts on the data generating process, so that the classic off-policy evaluation (OPE) estimators may fail to identify the true value function of the target policy. In this paper, we study the statistical properties of OPE in confounded MDPs with observable instrumental variables. Specifically, we propose a two-stage estimator based on the instrumental variables and establish its statistical properties in the confounded MDPs with a linear structure. For non-asymptotic analysis, we prove a $\mathcal{O}(n^{-1/2})$-error bound where $n$ is the number of samples. For asymptotic analysis, we prove that the two-stage estimator is asymptotically normal with a typical rate of $n^{1/2}$. To the best of our knowledge, we are the first to show such statistical results of the two-stage estimator for confounded linear MDPs via instrumental variables.

📄 PDF Abstract BibTeX arXiv:2209.05186

Code (0)

등록된 구현이 없습니다.

Tasks

Off-policy evaluation

Offline Reinforcement Learning with Instrumental Variables in Confounded Markov Decision Processes

2022-09-18 · Zuyue Fu, Zhengling Qi, Zhaoran Wang, Zhuoran Yang 외

We study the offline reinforcement learning (RL) in the face of unmeasured confounders. Due to the lack of online interaction with the environment, offline RL is facing the following two significant challenges: (i) the a…

Offline RL reinforcement-learning Reinforcement Learning (RL)

Kernel Instrumental Variable Regression

2019-06-01 · NeurIPS 2019 12 · Rahul Singh, Maneesh Sahani, Arthur Gretton

Instrumental variable (IV) regression is a strategy for learning causal relationships in observational data. If measurements of input X and output Y are confounded, the causal relationship can nonetheless be identified i…

regression

An Instrumental Variable Approach to Confounded Off-Policy Evaluation

2022-12-29 · Yang Xu, Jin Zhu, Chengchun Shi, Shikai Luo 외

Off-policy evaluation (OPE) is a method for estimating the return of a target policy using some pre-collected observational data generated by a potentially different behavior policy. In some cases, there may be unmeasure…

Decision Making Off-policy evaluation

Dual Instrumental Method for Confounded Kernelized Bandits

2022-09-07 · Xueping Gong, Jiheng Zhang

The contextual bandit problem is a theoretically justified framework with wide applications in various fields. While the previous study on this problem usually requires independence between noise and contexts, our work c…

Instrumental Variable Value Iteration for Causal Offline Reinforcement Learning

2021-02-19 · Luofeng Liao, Zuyue Fu, Zhuoran Yang, Yixin Wang 외

In offline reinforcement learning (RL) an optimal policy is learned solely from a priori collected observational data. However, in observational data, actions are often confounded by unobserved variables. Instrumental va…

Offline RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1