Act-Then-Measure: Reinforcement Learning for Partially Observable Environments with Active Measuring
We study Markov decision processes (MDPs), where agents have direct control over when and how they gather information, as formalized by action-contingent noiselessly observable MDPs (ACNO-MPDs). In these models, actions consist of two components: a control action that affects the environment, and a measurement action that affects what the agent can observe. To solve ACNO-MDPs, we introduce the act-then-measure (ATM) heuristic, which assumes that we can ignore future state uncertainty when choosing control actions. We show how following this heuristic may lead to shorter policy computation times and prove a bound on the performance loss incurred by the heuristic. To decide whether or not to take a measurement action, we introduce the concept of measuring value. We develop a reinforcement learning algorithm based on the ATM heuristic, using a Dyna-Q variant adapted for partially observable domains, and showcase its superior performance compared to prior methods on a number of partially-observable environments.
Code (1)
Tasks
reinforcement-learningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Future Prediction Can be a Strong Evidence of Good History Representation in Partially Observable Environments
Learning a good history representation is one of the core challenges of reinforcement learning (RL) in partially observable environments. Recent works have shown the advantages of various auxiliary tasks for facilitating…
Future predictionMemorizationPredictionreinforcement-learning+3Provably Efficient Partially Observable Risk-Sensitive Reinforcement Learning with Hindsight Observation
This work pioneers regret analysis of risk-sensitive reinforcement learning in partially observable environments with hindsight observation, addressing a gap in theoretical exploration. We introduce a novel formulation t…
reinforcement-learningReinforcement LearningOn Improving Deep Reinforcement Learning for POMDPs
Deep Reinforcement Learning (RL) recently emerged as one of the most competitive approaches for learning in sequential decision making problems with fully observable environments, e.g., computer Go. However, very little …
Atari GamesDecision MakingDeep Reinforcement Learningreinforcement-learning+5On Improving Deep Reinforcement Learning for POMDPs
Deep Reinforcement Learning (RL) recently emerged as one of the most competitive approaches for learning in sequential decision making problems with fully observable environments, e.g., computer Go. However, very little …
Atari GamesDecision MakingDeep Reinforcement Learningreinforcement-learning+5Policy Gradient in Partially Observable Environments: Approximation and Convergence
Policy gradient is a generic and flexible reinforcement learning approach that generally enjoys simplicity in analysis, implementation, and deployment. In the last few decades, this approach has been extensively advanced…
Decision MakingPolicy Gradient MethodsReinforcement Learning