Off-Policy Evaluation in Partially Observable Environments
This work studies the problem of batch off-policy evaluation for Reinforcement Learning in partially observable environments. Off-policy evaluation under partial observability is inherently prone to bias, with risk of arbitrarily large errors. We define the problem of off-policy evaluation for Partially Observable Markov Decision Processes (POMDPs) and establish what we believe is the first off-policy evaluation result for POMDPs. In addition, we formulate a model in which observed and unobserved variables are decoupled into two dynamic processes, called a Decoupled POMDP. We show how off-policy evaluation can be performed under this new model, mitigating estimation errors inherent to general POMDPs. We demonstrate the pitfalls of off-policy evaluation in POMDPs using a well-known off-policy method, Importance Sampling, and compare it with our result on synthetic medical data.
Code (0)
등록된 구현이 없습니다.
Tasks
Off-policy evaluationReinforcement LearningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Policy Gradient in Partially Observable Environments: Approximation and Convergence
Policy gradient is a generic and flexible reinforcement learning approach that generally enjoys simplicity in analysis, implementation, and deployment. In the last few decades, this approach has been extensively advanced…
Decision MakingPolicy Gradient MethodsReinforcement LearningHierarchical and Partially Observable Goal-driven Policy Learning with Goals Relational Graph
We present a novel two-layer hierarchical reinforcement learning approach equipped with a Goals Relational Graph (GRG) for tackling the partially observable goal-driven task, such as goal-driven visual navigation. Our GR…
Hierarchical Reinforcement LearningReinforcement Learning (RL)Visual NavigationPartially Observable Reference Policy Programming: Solving POMDPs Sans Numerical Optimisation
This paper proposes Partially Observable Reference Policy Programming, a novel anytime online approximate POMDP solver which samples meaningful future histories very deeply while simultaneously forcing a gradual policy u…
Future Prediction Can be a Strong Evidence of Good History Representation in Partially Observable Environments
Learning a good history representation is one of the core challenges of reinforcement learning (RL) in partially observable environments. Recent works have shown the advantages of various auxiliary tasks for facilitating…
Future predictionMemorizationPredictionreinforcement-learning+3Learning to Cooperate via Policy Search
Cooperative games are those in which both agents share the same payoff structure. Value-based reinforcement-learning algorithms, such as variants of Q-learning, have been applied to learning cooperative games, but they o…
Q-Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)