Steady State Analysis of Episodic Reinforcement Learning

2020-11-12 · NeurIPS 2020 12 · Huang Bojun

This paper proves that the episodic learning environment of every finite-horizon decision task has a unique steady state under any behavior policy, and that the marginal distribution of the agent's input indeed converges to the steady-state distribution in essentially all episodic learning processes. This observation supports an interestingly reversed mindset against conventional wisdom: While the existence of unique steady states was often presumed in continual learning but considered less relevant in episodic learning, it turns out their existence is guaranteed for the latter. Based on this insight, the paper unifies episodic and continual RL around several important concepts that have been separately treated in these two RL formalisms. Practically, the existence of unique and approachable steady state enables a general way to collect data in episodic RL tasks, which the paper applies to policy gradient algorithms as a demonstration, based on a new steady-state policy gradient theorem. Finally, the paper also proposes and experimentally validates a perturbation method that facilitates rapid steady-state convergence in real-world RL tasks.

📄 PDF Abstract BibTeX arXiv:2011.06631

Code (0)

등록된 구현이 없습니다.

Tasks

Continual Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Neural Episodic Control with State Abstraction

2023-01-27 · Zhuo Li, Derui Zhu, Yujing Hu, Xiaofei Xie 외

Existing Deep Reinforcement Learning (DRL) algorithms suffer from sample inefficiency. Generally, episodic control-based approaches are solutions that leverage highly-rewarded past experiences to improve sample efficienc…

Deep Reinforcement Learning MuJoCo OpenAI Gym

Episodic Reinforcement Learning with Associative Memory

2020-05-01 · ICLR 2020 1 · Guangxiang Zhu*, Zichuan Lin*, Guangwen Yang, Chongjie Zhang

Sample efficiency has been one of the major challenges for deep reinforcement learning. Non-parametric episodic control has been proposed to speed up parametric reinforcement learning by rapidly latching on previously su…

Atari Games Deep Reinforcement Learning reinforcement-learning Reinforcement Learning+1

Two-Memory Reinforcement Learning

2023-04-20 · Zhao Yang, Thomas. M. Moerland, Mike Preuss, Aske Plaat

While deep reinforcement learning has shown important empirical success, it tends to learn relatively slow due to slow propagation of rewards information and slow update of parametric neural networks. Non-parametric epis…

Deep Reinforcement Learning reinforcement-learning Reinforcement Learning Representation Learning+1

Constrained episodic reinforcement learning in concave-convex and knapsack settings

2020-06-09 · NeurIPS 2020 12 · Kianté Brantley, Miroslav Dudik, Thodoris Lykouris, Sobhan Miryoosefi 외

We propose an algorithm for tabular episodic reinforcement learning with constraints. We provide a modular analysis with strong theoretical guarantees for settings with concave rewards and convex constraints, and for set…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Model-Free Episodic Control with State Aggregation

2020-08-21 · Rafael Pinto

Episodic control provides a highly sample-efficient method for reinforcement learning while enforcing high memory and computational requirements. This work proposes a simple heuristic for reducing these requirements, and…

Atari Games model reinforcement-learning Reinforcement Learning+1