ISC-POMDPs: Partially Observed Markov Decision Processes with Initial-State Dependent Costs
We introduce a class of partially observed Markov decision processes (POMDPs) with costs that can depend on both the value and (future) uncertainty associated with the initial state. These Initial-State Cost POMDPs (ISC-POMDPs) enable the specification of objectives relative to a priori unknown initial states, which is useful in applications such as robot navigation, controlled sensing, and active perception, that can involve controlling systems to revisit, remain near, or actively infer their initial states. By developing a recursive Bayesian fixed-point smoother to estimate the initial state that resembles the standard recursive Bayesian filter, we show that ISC-POMDPs can be treated as POMDPs with (potentially) belief-dependent costs. We demonstrate the utility of ISC-POMDPs, including their ability to select controls that resolve (future) uncertainty about (past) initial states, in simulation.
Code (0)
등록된 구현이 없습니다.
Tasks
Robot NavigationSimilar Papers 제목 키워드 기반
Entropy-Regularized Partially Observed Markov Decision Processes
We investigate partially observed Markov decision processes (POMDPs) with cost functions regularized by entropy terms describing state, observation, and control uncertainty. Standard POMDP techniques are shown to offer b…
State EstimationOff-Policy Evaluation in Partially Observable Environments
This work studies the problem of batch off-policy evaluation for Reinforcement Learning in partially observable environments. Off-policy evaluation under partial observability is inherently prone to bias, with risk of ar…
Off-policy evaluationReinforcement LearningReinforcement Learning (RL)Incremental Pruning: A Simple, Fast, Exact Method for Partially Observable Markov Decision Processes
Most exact algorithms for general partially observable Markov decision processes (POMDPs) use a form of dynamic programming in which a piecewise-linear and convex representation of one value function is transformed into …
Off-Policy Evaluation in Partially Observed Markov Decision Processes under Sequential Ignorability
We consider off-policy evaluation of dynamic treatment rules under sequential ignorability, given an assumption that the underlying system can be modeled as a partially observed Markov decision process (POMDP). We propos…
Off-policy evaluationApproximation Methods for Partially Observed Markov Decision Processes (POMDPs)
POMDPs are useful models for systems where the true underlying state is not known completely to an outside observer; the outside observer incompletely knows the true state of the system, and observes a noisy version of t…
Survey