Flow-based Recurrent Belief State Learning for POMDPs
Partially Observable Markov Decision Process (POMDP) provides a principled and generic framework to model real world sequential decision making processes but yet remains unsolved, especially for high dimensional continuous space and unknown models. The main challenge lies in how to accurately obtain the belief state, which is the probability distribution over the unobservable environment states given historical information. Accurately calculating this belief state is a precondition for obtaining an optimal policy of POMDPs. Recent advances in deep learning techniques show great potential to learn good belief states. However, existing methods can only learn approximated distribution with limited flexibility. In this paper, we introduce the \textbf{F}l\textbf{O}w-based \textbf{R}ecurrent \textbf{BE}lief \textbf{S}tate model (FORBES), which incorporates normalizing flows into the variational inference to learn general continuous belief states for POMDPs. Furthermore, we show that the learned belief states can be plugged into downstream RL algorithms to improve performance. In experiments, we show that our methods successfully capture the complex belief states that enable multi-modal predictions as well as high quality reconstructions, and results on challenging visual-motor control tasks show that our method achieves superior performance and sample efficiency.
Code (0)
등록된 구현이 없습니다.
Tasks
Decision MakingSequential Decision MakingVariational InferenceMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Learning Belief Representations for Imitation Learning in POMDPs
We consider the problem of imitation learning from expert demonstrations in partially observable Markov decision processes (POMDPs). Belief representations, which characterize the distribution over the latent states in a…
continuous-controlContinuous ControlImitation LearningReinforcement Learning+1Optimizing Sequential Medical Treatments with Auto-Encoding Heuristic Search in POMDPs
Health-related data is noisy and stochastic in implying the true physiological states of patients, limiting information contained in single-moment observations for sequential clinical decision making. We model patient-cl…
Decision MakingHeuristic SearchPeriodic agent-state based Q-learning for POMDPs
The standard approach for Partially Observable Markov Decision Processes (POMDPs) is to convert them to a fully observed belief-state MDP. However, the belief state depends on the system model and is therefore not viable…
Q-LearningReinforcement Learning (RL)Rethinking Transformers in Solving POMDPs
Sequential decision-making algorithms such as reinforcement learning (RL) in real-world scenarios inevitably face environments with partial observability. This paper scrutinizes the effectiveness of a popular architectur…
Decision MakingReinforcement Learning (RL)Sequential Decision MakingMonte Carlo Information-Oriented Planning
In this article, we discuss how to solve information-gathering problems expressed as rho-POMDPs, an extension of Partially Observable Markov Decision Processes (POMDPs) whose reward rho depends on the belief state. Point…