A Theory of Abstraction in Reinforcement Learning

2022-03-01 · David Abel

Reinforcement learning defines the problem facing agents that learn to make good decisions through action and observation alone. To be effective problem solvers, such agents must efficiently explore vast worlds, assign credit from delayed feedback, and generalize to new experiences, all while making use of limited data, computational resources, and perceptual bandwidth. Abstraction is essential to all of these endeavors. Through abstraction, agents can form concise models of their environment that support the many practices required of a rational, adaptive decision maker. In this dissertation, I present a theory of abstraction in reinforcement learning. I first offer three desiderata for functions that carry out the process of abstraction: they should 1) preserve representation of near-optimal behavior, 2) be learned and constructed efficiently, and 3) lower planning or learning time. I then present a suite of new algorithms and analysis that clarify how agents can learn to abstract according to these desiderata. Collectively, these results provide a partial path toward the discovery and use of abstraction that minimizes the complexity of effective reinforcement learning.

📄 PDF Abstract BibTeX arXiv:2203.00397

Code (0)

등록된 구현이 없습니다.

Tasks

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

State Abstractions for Lifelong Reinforcement Learning

2018-07-01 · ICML 2018 7 · David Abel, Dilip Arumugam, Lucas Lehnert, Michael Littman

In lifelong reinforcement learning, agents must effectively transfer knowledge across tasks while simultaneously addressing exploration, credit assignment, and generalization. State abstraction can help overcome the…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Learning Abstractions for Hierarchical Planning in Program-Synthesis Agents

2026-01-31 · Zergham Ahmed, Kazuki Irie, Joshua B. Tenenbaum, Christopher J. Bates 외 arxiv

Humans learn abstractions and use them to plan efficiently to quickly generalize across tasks -- an ability that remains challenging for state-of-the-art large language model (LLM) agents and deep reinforcement learning …

Reinforcement Learning

Learning State Abstractions for Transfer in Continuous Control

2020-02-08 · Kavosh Asadi, David Abel, Michael L. Littman

Can simple algorithms with a good representation solve challenging reinforcement learning problems? In this work, we answer this question in the affirmative, where we take "simple learning algorithm" to be tabular Q-Lear…

continuous-control Continuous Control Q-Learning reinforcement-learning+2

Distributional Active Inference

2026-01-28 · Abdullah Akgül, Gulcin Baykal, Manuel Haußmann, Mustafa Mert Çelikok 외 arxiv

Optimal control of complex environments with robotic systems faces two complementary and intertwined challenges: efficient organization of sensory state information and far-sighted action planning. Because the reinforcem…

Reinforcement Learning

Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning

1999-08-06 · Artificial Intelligence 1999 8 · Richard S. Sutton, Doina Precup, Satinder Singh

Learning, planning, and representing knowledge at multiple levels of temporal abstraction are key, longstanding challenges for AI. In this paper we consider how these challenges can be addressed within the mathematical f…

Q-Learning reinforcement-learning Reinforcement Learning