Off-policy Evaluation with Deeply-abstracted States
Off-policy evaluation (OPE) is crucial for assessing a target policy's impact offline before its deployment. However, achieving accurate OPE in large state spaces remains challenging. This paper studies state abstractions -- originally designed for policy learning -- in the context of OPE. Our contributions are three-fold: (i) We define a set of irrelevance conditions central to learning state abstractions for OPE, and derive a backward-model-irrelevance condition for achieving irrelevance in %sequential and (marginalized) importance sampling ratios by constructing a time-reversed Markov decision process (MDP). (ii) We propose a novel iterative procedure that sequentially projects the original state space into a smaller space, resulting in a deeply-abstracted state, which substantially simplifies the sample complexity of OPE arising from high cardinality. (iii) We prove the Fisher consistencies of various OPE estimators when applied to our proposed abstract state spaces.
Code (1)
Tasks
Off-policy evaluationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Generation of Policy-Level Explanations for Reinforcement Learning
Though reinforcement learning has greatly benefited from the incorporation of neural networks, the inability to verify the correctness of such systems limits their use. Current work in explainable deep learning focuses o…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Deeply-Debiased Off-Policy Interval Estimation
Off-policy evaluation learns a target policy's value with a historical dataset generated by a different behavior policy. In addition to a point estimate, many applications would benefit significantly from having a confid…
Off-policy evaluationAUPO -- Abstracted Until Proven Otherwise: A Reward Distribution Based Abstraction Algorithm
We introduce a novel, drop-in modification to Monte Carlo Tree Search's (MCTS) decision policy that we call AUPO. Comparisons based on a range of IPPC benchmark problems show that AUPO clearly outperforms MCTS. AUPO is a…
Visual Relationship Detection with Relative Location Mining
Visual relationship detection, as a challenging task used to find and distinguish the interactions between object pairs in one image, has received much attention recently. In this work, we propose a novel visual relation…
Graph Neural NetworkObjectRelationship DetectionVisual Relationship DetectionUnveiling the Potential of Graph Neural Networks in SME Credit Risk Assessment
This paper takes the graph neural network as the technical framework, integrates the intrinsic connections between enterprise financial indicators, and proposes a model for enterprise credit risk assessment. The main res…
Graph Neural NetworkRepresentation Learning