CoinDICE: Off-Policy Confidence Interval Estimation
We study high-confidence behavior-agnostic off-policy evaluation in reinforcement learning, where the goal is to estimate a confidence interval on a target policy's value, given only access to a static experience dataset collected by unknown behavior policies. Starting from a function space embedding of the linear program formulation of the $Q$-function, we obtain an optimization problem with generalized estimating equation constraints. By applying the generalized empirical likelihood method to the resulting Lagrangian, we propose CoinDICE, a novel and efficient algorithm for computing confidence intervals. Theoretically, we prove the obtained confidence intervals are valid, in both asymptotic and finite-sample regimes. Empirically, we show in a variety of benchmarks that the confidence interval estimates are tighter and more accurate than existing methods.
Code (0)
등록된 구현이 없습니다.
Tasks
Off-policy evaluationReinforcement Learning (RL)validSimilar Papers 제목 키워드 기반
Accountable Off-Policy Evaluation via a Kernelized Bellman Statistics
Off-policy evaluation plays an important role in modern reinforcement learning. However, most of the existing off-policy evaluation only focus on the value estimation, without providing an accountable confidence interval…
Off-policy evaluationAsymptotically Unbiased Off-Policy Policy Evaluation when Reusing Old Data in Nonstationary Environments
In this work, we consider the off-policy policy evaluation problem for contextual bandits and finite horizon reinforcement learning in the nonstationary setting. Reusing old data is critical for policy evaluation, but ex…
Multi-Armed BanditsregressionSurvey SamplingvalidStatistical Bootstrapping for Uncertainty Estimation in Off-Policy Evaluation
In reinforcement learning, it is typical to use the empirically observed transitions and rewards to estimate the value of a policy via either model-based or Q-fitting approaches. Although straightforward, these technique…
continuous-controlContinuous ControlOff-policy evaluationInterval Estimation for Reinforcement-Learning Algorithms in Continuous-State Domains
The reinforcement learning community has explored many approaches to obtain- ing value estimates and models to guide decision making; these approaches, how- ever, do not usually provide a measure of confidence in the est…
Decision Makingparameter estimationreinforcement-learningReinforcement Learning+1Empirical Likelihood for Contextual Bandits
We propose an estimator and confidence interval for computing the value of a policy from off-policy data in the contextual bandit setting. To this end we apply empirical likelihood techniques to formulate our estimator a…
Multi-Armed Bandits