Provable Rich Observation Reinforcement Learning with Combinatorial Latent States

2021-01-01 · ICLR 2021 1 · Dipendra Misra, Qinghua Liu, Chi Jin, John Langford

We propose a novel setting for reinforcement learning that combines two common real-world difficulties: presence of observations (such as camera images) and factored states (such as location of objects). In our setting, the agent receives observations generated stochastically from a "latent" factored state. These observations are "rich enough" to enable decoding of the latent state and remove partial observability concerns. Since the latent state is combinatorial, the size of state space is exponential in the number of latent factors. We create a learning algorithm FactoRL (Fact-o-Rel) for this setting, which uses noise-contrastive learning to identify latent structures in emission processes and discover a factorized state space. We derive polynomial sample complexity guarantees for FactoRL which polynomially depend upon the number factors, and very weakly depend on the size of the observation space. We also provide a guarantee of polynomial time complexity when given access to an efficient planning algorithm.

📄 PDF Abstract BibTeX

Code (0)

등록된 구현이 없습니다.

Tasks

Contrastive Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Extracting Latent State Representations with Linear Dynamics from Rich Observations

2020-06-29 · Abraham Frandsen, Rong Ge

Recently, many reinforcement learning techniques were shown to have provable guarantees in the simple case of linear dynamics, especially in problems like linear quadratic regulators. However, in practice, many reinforce…

Position reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Provable Reinforcement Learning with a Short-Term Memory

2022-02-08 · Yonathan Efroni, Chi Jin, Akshay Krishnamurthy, Sobhan Miryoosefi

Real-world sequential decision making problems commonly involve partial observability, which requires the agent to maintain a memory of history in order to infer the latent states, plan and make good decisions. Coping wi…

Decision Making reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1

Learning the Linear Quadratic Regulator from Nonlinear Observations

2020-10-08 · NeurIPS 2020 12 · Zakaria Mhammedi, Dylan J. Foster, Max Simchowitz, Dipendra Misra 외

We introduce a new problem setting for continuous control called the LQR with Rich Observations, or RichLQR. In our setting, the environment is summarized by a low-dimensional continuous latent state with linear dynamics…

continuous-control Continuous Control Decoder

Provable RL with Exogenous Distractors via Multistep Inverse Dynamics

2021-10-17 · Yonathan Efroni, Dipendra Misra, Akshay Krishnamurthy, Alekh Agarwal 외

Many real-world applications of reinforcement learning (RL) require the agent to deal with high-dimensional observations such as those generated from a megapixel camera. Prior work has addressed such problems with repres…

Reinforcement Learning (RL)Representation Learning

Reinforcement Learning under Latent Dynamics: Toward Statistical and Algorithmic Modularity

2024-10-23 · Philip Amortila, Dylan J. Foster, Nan Jiang, Akshay Krishnamurthy 외

Real-world applications of reinforcement learning often involve environments where agents operate on complex, high-dimensional observations, but the underlying (''latent'') dynamics are comparatively simple. However, out…

reinforcement-learning Reinforcement Learning