Approximate information state for approximate planning and reinforcement learning in partially observed systems

2020-10-17 · Jayakumar Subramanian, Amit Sinha, Raihan Seraj, Aditya Mahajan

We propose a theoretical framework for approximate planning and learning in partially observed systems. Our framework is based on the fundamental notion of information state. We provide two equivalent definitions of information state -- i) a function of history which is sufficient to compute the expected reward and predict its next value; ii) equivalently, a function of the history which can be recursively updated and is sufficient to compute the expected reward and predict the next observation. An information state always leads to a dynamic programming decomposition. Our key result is to show that if a function of the history (called approximate information state (AIS)) approximately satisfies the properties of the information state, then there is a corresponding approximate dynamic program. We show that the policy computed using this is approximately optimal with bounded loss of optimality. We show that several approximations in state, observation and action spaces in literature can be viewed as instances of AIS. In some of these cases, we obtain tighter bounds. A salient feature of AIS is that it can be learnt from data. We present AIS based multi-time scale policy gradient algorithms. and detailed numerical experiments with low, moderate and high dimensional environments.

📄 PDF Abstract BibTeX arXiv:2010.08843

Code (1)

info-structures/ais 공식 구현 pytorch

Tasks

reinforcement-learning Reinforcement Learning (RL)

Near Optimal Behavior via Approximate State Abstraction

2017-01-15 · David Abel, D. Ellis Hershkowitz, Michael L. Littman

The combinatorial explosion that plagues planning and reinforcement learning (RL) algorithms can be moderated using state abstraction. Prohibitively large task representations can be condensed such that essential informa…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Planning to the Information Horizon of BAMDPs via Epistemic State Abstraction

2022-10-30 · Dilip Arumugam, Satinder Singh

The Bayes-Adaptive Markov Decision Process (BAMDP) formalism pursues the Bayes-optimal solution to the exploration-exploitation trade-off in reinforcement learning. As the computation of exact solutions to Bayesian reinf…

Efficient Exploration reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Hierarchical Reinforcement Learning: Approximating Optimal Discounted TSP Using Local Policies

2018-03-13 · Tom Zahavy, Avinatan Hasidim, Haim Kaplan, Yishay Mansour

In this work, we provide theoretical guarantees for reward decomposition in deterministic MDPs. Reward decomposition is a special case of Hierarchical Reinforcement Learning, that allows one to learn many policies in par…

Hierarchical Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1

Learning to Execute: Efficient Learning of Universal Plan-Conditioned Policies in Robotics

2021-11-15 · NeurIPS 2021 12 · Ingmar Schubert, Danny Driess, Ozgur S. Oguz, Marc Toussaint

Applications of Reinforcement Learning (RL) in robotics are often limited by high data demand. On the other hand, approximate models are readily available in many robotics scenarios, making model-based approaches like pl…

Learning to Execute Reinforcement Learning (RL)

Fast Exploration with Simplified Models and Approximately Optimistic Planning in Model Based Reinforcement Learning

2018-06-01 · Ramtin Keramati, Jay Whang, Patrick Cho, Emma Brunskill

Humans learn to play video games significantly faster than the state-of-the-art reinforcement learning (RL) algorithms. People seem to build simple models that are easy to learn to support planning and strategic explorat…

Model-based Reinforcement Learning Object reinforcement-learning Reinforcement Learning+1