Asymptotically Exact Error Characterization of Offline Policy Evaluation with Misspecified Linear Models

2021-12-01 · NeurIPS 2021 12 · Kohei Miyaguchi

We consider the problem of offline policy evaluation~(OPE) with Markov decision processes~(MDPs), where the goal is to estimate the utility of given decision-making policies based on static datasets. Recently, theoretical understanding of OPE has been rapidly advanced under (approximate) realizability assumptions, i.e., where the environments of interest are well approximated with the given hypothetical models. On the other hand, the OPE under unrealizability has not been well understood as much as in the realizable setting despite its importance in real-world applications.To address this issue, we study the behavior of a simple existing OPE method called the linear direct method~(DM) under the unrealizability. Consequently, we obtain an asymptotically exact characterization of the OPE error in a doubly robust form. Leveraging this result, we also establish the nonparametric consistency of the tile-coding estimators under quite mild assumptions.

📄 PDF Abstract BibTeX

Code (0)

등록된 구현이 없습니다.

Tasks

Decision Making

Optimal Sensor Collaboration for Parameter Tracking Using Energy Harvesting Sensors

2018-03-25

In this paper, we design an optimal sensor collaboration strategy among neighboring nodes while tracking a time-varying parameter using wireless sensor networks in the presence of imperfect communication channels. The se…

Asymptotically Efficient Off-Policy Evaluation for Tabular Reinforcement Learning

2020-01-29 · Ming Yin, Yu-Xiang Wang

We consider the problem of off-policy evaluation for reinforcement learning, where the goal is to estimate the expected reward of a target policy $\pi$ using offline data collected by running a logging policy $\mu$. Stan…

Off-policy evaluation reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Optimal Learning for Sequential Decision Making for Expensive Cost Functions with Stochastic Binary Feedbacks

2017-09-13 · Yingfei Wang, Chu Wang, Warren Powell

We consider the problem of sequentially making decisions that are rewarded by "successes" and "failures" which can be predicted through an unknown relationship that depends on a partially controllable vector of attribute…

Decision Making Multi-Armed Bandits Sequential Decision Making

Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning

2022-02-23 · ICLR 2022 4 · Chenjia Bai, Lingxiao Wang, Zhuoran Yang, Zhihong Deng 외

Offline Reinforcement Learning (RL) aims to learn policies from previously collected datasets without exploring the environment. Directly applying off-policy algorithms to offline RL usually fails due to the extrapolatio…

D4RL Offline RL reinforcement-learning Reinforcement Learning+2

On the Optimal Sample Complexity of Offline Multi-Armed Bandits with KL Regularization

2026-05-04 · Kaixuan Ji, Qiwei Di, Heyang Zhao, Qingyue Zhao 외 arxiv

Kullback-Leibler (KL) regularization is widely used in offline decision-making and offers several benefits, motivating recent work on the sample complexity of offline learning with respect to KL-regularized performance m…

Multi-Armed Bandits