Minimax Value Interval for Off-Policy Evaluation and Policy Optimization
We study minimax methods for off-policy evaluation (OPE) using value functions and marginalized importance weights. Despite that they hold promises of overcoming the exponential variance in traditional importance sampling, several key problems remain: (1) They require function approximation and are generally biased. For the sake of trustworthy OPE, is there anyway to quantify the biases? (2) They are split into two styles ("weight-learning" vs "value-learning"). Can we unify them? In this paper we answer both questions positively. By slightly altering the derivation of previous methods (one from each style; Uehara et al., 2020), we unify them into a single value interval that comes with a special type of double robustness: when either the value-function or the importance-weight class is well specified, the interval is valid and its length quantifies the misspecification of the other class. Our interval also provides a unified view of and new insights to some recent methods, and we further explore the implications of our results on exploration and exploitation in off-policy policy optimization with insufficient data coverage.
Code (0)
등록된 구현이 없습니다.
Tasks
Efficient ExplorationOff-policy evaluationvalidSimilar Papers 제목 키워드 기반
Using Ordinal Voting to Compare the Utilitarian Welfare of a Status Quo and A Proposed Policy: A Simple Nonparametric Analysis
The relationship of policy choice by majority voting and by maximization of utilitarian welfare has long been discussed. I consider choice between a status quo and a proposed policy when persons have interpersonally comp…
On Minimax Optimal Offline Policy Evaluation
This paper studies the off-policy evaluation problem, where one aims to estimate the value of a target policy based on a sample of observations collected by another policy. We first consider the multi-armed bandit case, …
Multi-Armed BanditsOff-policy evaluationAccountable Off-Policy Evaluation via a Kernelized Bellman Statistics
Off-policy evaluation plays an important role in modern reinforcement learning. However, most of the existing off-policy evaluation only focus on the value estimation, without providing an accountable confidence interval…
Off-policy evaluationBootstrapping Fitted Q-Evaluation for Off-Policy Inference
Bootstrapping provides a flexible and effective approach for assessing the quality of batch reinforcement learning, yet its theoretical property is less understood. In this paper, we study the use of bootstrapping in off…
Off-policy evaluationDeeply-Debiased Off-Policy Interval Estimation
Off-policy evaluation learns a target policy's value with a historical dataset generated by a different behavior policy. In addition to a point estimate, many applications would benefit significantly from having a confid…
Off-policy evaluation