Accountable Off-Policy Evaluation With Kernel Bellman Statistics
We consider off-policy evaluation (OPE), which evaluates the performance of a new policy from observed data collected from previous experiments, without requiring the execution of the new policy. This finds important applications in areas with high execution cost or safety concerns, such as medical diagnosis, recommendation systems and robotics. In practice, due to the limited information from off-policy data, it is highly desirable to construct rigorous confidence intervals, not just point estimation, for the policy performance. In this work, we propose a new variational framework which reduces the problem of calculating tight confidence bounds in OPE into an optimization problem on a feasible set that catches the true state-action value function with high probability. The feasible set is constructed by leveraging statistical properties of a recently proposed kernel Bellman loss (Feng et al., 2019). We design an efficient computational approach for calculating our bounds, and extend it to perform post-hoc diagnosis and correction for existing estimators. Empirical results show that our method yields tight confidence intervals in different settings.
Code (0)
등록된 구현이 없습니다.
Tasks
Medical DiagnosisOff-policy evaluationRecommendation SystemsSimilar Papers 제목 키워드 기반
Accountable Off-Policy Evaluation via a Kernelized Bellman Statistics
Off-policy evaluation plays an important role in modern reinforcement learning. However, most of the existing off-policy evaluation only focus on the value estimation, without providing an accountable confidence interval…
Off-policy evaluationNonparametric Bellman Mappings for Reinforcement Learning: Application to Robust Adaptive Filtering
This paper designs novel nonparametric Bellman mappings in reproducing kernel Hilbert spaces (RKHSs) for reinforcement learning (RL). The proposed mappings benefit from the rich approximating properties of RKHSs, adopt n…
Dimensionality ReductionReinforcement Learning (RL)Vector-Valued Distributional Reinforcement Learning Policy Evaluation: A Hilbert Space Embedding Approach
We propose an (offline) multi-dimensional distributional reinforcement learning framework (KE-DRL) that leverages Hilbert space mappings to estimate the kernel mean embedding of the multi-dimensional value distribution u…
Reinforcement LearningWhy Should I Trust You, Bellman? Evaluating the Bellman Objective with Off-Policy Data
In this work, we analyze the effectiveness of the Bellman equation as a proxy objective for value prediction accuracy in off-policy evaluation. While the Bellman equation is uniquely solved by the true value function ove…
Deep Reinforcement LearningOff-policy evaluationValue predictionOptimal policy evaluation using kernel-based temporal difference methods
We study methods based on reproducing kernel Hilbert spaces for estimating the value function of an infinite-horizon discounted Markov reward process (MRP). We study a regularized form of the kernel least-squares tempora…