Sample Efficient Reinforcement Learning In Continuous State Spaces: A Perspective Beyond Linearity

2021-06-15 · Dhruv Malik, Aldo Pacchiano, Vishwak Srinivasan, Yuanzhi Li

Reinforcement learning (RL) is empirically successful in complex nonlinear Markov decision processes (MDPs) with continuous state spaces. By contrast, the majority of theoretical RL literature requires the MDP to satisfy some form of linear structure, in order to guarantee sample efficient RL. Such efforts typically assume the transition dynamics or value function of the MDP are described by linear functions of the state features. To resolve this discrepancy between theory and practice, we introduce the Effective Planning Window (EPW) condition, a structural condition on MDPs that makes no linearity assumptions. We demonstrate that the EPW condition permits sample efficient RL, by providing an algorithm which provably solves MDPs satisfying this condition. Our algorithm requires minimal assumptions on the policy class, which can include multi-layer neural networks with nonlinear activation functions. Notably, the EPW condition is directly motivated by popular gaming benchmarks, and we show that many classic Atari games satisfy this condition. We additionally show the necessity of conditions like EPW, by demonstrating that simple MDPs with slight nonlinearities cannot be solved sample efficiently.

📄 PDF Abstract BibTeX arXiv:2106.07814

Code (0)

등록된 구현이 없습니다.

Tasks

Atari Games reinforcement-learning Reinforcement Learning (RL)

Taming "data-hungry" reinforcement learning? Stability in continuous state-action spaces

2024-01-10 · Yaqi Duan, Martin J. Wainwright

We introduce a novel framework for analyzing reinforcement learning (RL) in continuous state-action spaces, and use it to prove fast rates of convergence in both off-line and on-line settings. Our analysis highlights two…

reinforcement-learning Reinforcement Learning (RL)Transfer Learning

DPO: A Differential and Pointwise Control Approach to Reinforcement Learning

2024-04-24 · Minh Nguyen, Chandrajit Bajaj

Reinforcement learning (RL) in continuous state-action spaces remains challenging in scientific computing due to poor sample efficiency and lack of pathwise physical consistency. We introduce Differential Reinforcement L…

Benchmarking reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Dream and Search to Control: Latent Space Planning for Continuous Control

2020-10-19 · Anurag Koul, Varun V. Kumar, Alan Fern, Somdeb Majumdar

Learning and planning with latent space dynamics has been shown to be useful for sample efficiency in model-based reinforcement learning (MBRL) for discrete and continuous control tasks. In particular, recent work, for d…

continuous-control Continuous Control Model-based Reinforcement Learning Reinforcement Learning (RL)

Continuous Motion Planning with Temporal Logic Specifications using Deep Neural Networks

2020-04-02 · Chuanzheng Wang, Yi-Nan Li, Stephen L. Smith, Jun Liu

In this paper, we propose a model-free reinforcement learning method to synthesize control policies for motion planning problems with continuous states and actions. The robot is modelled as a labeled discrete-time Markov…

Motion Planning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

The Sample Complexity of Online Reinforcement Learning: A Multi-model Perspective

2025-01-27 · Michael Muehlebach, Zhiyu He, Michael I. Jordan

We study the sample complexity of online reinforcement learning in the general setting of nonlinear dynamical systems with continuous state and action spaces. Our analysis accommodates a large class of dynamical systems …