Off-Policy Interval Estimation with Lipschitz Value Iteration
Off-policy evaluation provides an essential tool for evaluating the effects of different policies or treatments using only observed data. When applied to high-stakes scenarios such as medical diagnosis or financial decision-making, it is crucial to provide provably correct upper and lower bounds of the expected reward, not just a classical single point estimate, to the end-users, as executing a poor policy can be very costly. In this work, we propose a provably correct method for obtaining interval bounds for off-policy evaluation in a general continuous setting. The idea is to search for the maximum and minimum values of the expected reward among all the Lipschitz Q-functions that are consistent with the observations, which amounts to solving a constrained optimization problem on a Lipschitz function space. We go on to introduce a Lipschitz value iteration method to monotonically tighten the interval, which is simple yet efficient and provably convergent. We demonstrate the practical efficiency of our method on a range of benchmarks.
Code (0)
등록된 구현이 없습니다.
Tasks
Decision MakingMedical DiagnosisOff-policy evaluationSimilar Papers 제목 키워드 기반
A Contracting Dynamical System Perspective toward Interval Markov Decision Processes
Interval Markov decision processes are a class of Markov models where the transition probabilities between the states belong to intervals. In this paper, we study the problem of efficient estimation of the optimal polici…
Neural Policy Iteration for Stochastic Optimal Control: A Physics-Informed Approach
We propose a physics-informed neural network policy iteration (PINN-PI) framework for solving stochastic optimal control problems governed by second-order Hamilton--Jacobi--Bellman (HJB) equations. At each iteration, a n…
Efficient Bound of Lipschitz Constant for Convolutional Layers by Gram Iteration
Since the control of the Lipschitz constant has a great impact on the training stability, generalization, and robustness of neural networks, the estimation of this value is nowadays a real scientific challenge. In this p…
Accountable Off-Policy Evaluation via a Kernelized Bellman Statistics
Off-policy evaluation plays an important role in modern reinforcement learning. However, most of the existing off-policy evaluation only focus on the value estimation, without providing an accountable confidence interval…
Off-policy evaluationSolving nonconvex Hamilton--Jacobi--Isaacs equations with PINN-based policy iteration
We propose a mesh-free policy iteration framework that combines classical dynamic programming with physics-informed neural networks (PINNs) to solve high-dimensional, nonconvex Hamilton--Jacobi--Isaacs (HJI) equations ar…
Multi-agent Reinforcement Learning