Autonomous Option Invention for Continual Hierarchical Reinforcement Learning and Planning

2024-12-20 · Rashmeet Kaur Nayyar, Siddharth Srivastava

Abstraction is key to scaling up reinforcement learning (RL). However, autonomously learning abstract state and action representations to enable transfer and generalization remains a challenging open problem. This paper presents a novel approach for inventing, representing, and utilizing options, which represent temporally extended behaviors, in continual RL settings. Our approach addresses streams of stochastic problems characterized by long horizons, sparse rewards, and unknown transition and reward functions. Our approach continually learns and maintains an interpretable state abstraction, and uses it to invent high-level options with abstract symbolic representations. These options meet three key desiderata: (1) composability for solving tasks effectively with lookahead planning, (2) reusability across problem instances for minimizing the need for relearning, and (3) mutual independence for reducing interference among options. Our main contributions are approaches for continually learning transferable, generalizable options with symbolic representations, and for integrating search techniques with RL to efficiently plan over these learned options to solve new problems. Empirical results demonstrate that the resulting approach effectively learns and transfers abstract knowledge across problem instances, achieving superior sample efficiency compared to state-of-the-art methods.

📄 PDF Abstract BibTeX arXiv:2412.16395

Code (1)

aair-lab/chirp 공식 구현

Tasks

Hierarchical Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Matching options to tasks using Option-Indexed Hierarchical Reinforcement Learning

2022-06-12 · Kushal Chauhan, Soumya Chatterjee, Akash Reddy, Balaraman Ravindran 외

The options framework in Hierarchical Reinforcement Learning breaks down overall goals into a combination of options or simpler tasks and associated policies, allowing for abstraction in the action space. Ideally, these …

Continual Learning Hierarchical Reinforcement Learning reinforcement-learning Reinforcement Learning+2

Learning to Drive Safely with Hybrid Options

2025-10-28 · Bram De Cooman, Johan Suykens arxiv

Out of the many deep reinforcement learning approaches for autonomous driving, only few make use of the options (or skills) framework. That is surprising, as this framework is naturally suited for hierarchical control ap…

Reinforcement Learning Autonomous Driving

Hierarchical Subspaces of Policies for Continual Offline Reinforcement Learning

2024-12-19 · Anthony Kobanda, Rémy Portelas, Odalric-Ambrym Maillard, Ludovic Denoyer

In dynamic domains such as autonomous robotics and video game simulations, agents must continuously adapt to new tasks while retaining previously acquired skills. This ongoing process, known as Continual Reinforcement Le…

Continual Learning MuJoCo reinforcement-learning Reinforcement Learning

Generative AI at the Crossroads: Light Bulb, Dynamo, or Microscope?

2025-05-20 · Martin Baily, David Byrne, Aidan Kane, Paul Soto

With the advent of generative AI (genAI), the potential scope of artificial intelligence has increased dramatically, but the future effect of genAI on productivity remains uncertain with the effect of the technology on t…

Enhancing Hierarchical Reinforcement Learning through Change Point Detection in Time Series

2025-10-28 · Hemanath Arumugam, Falong Fan, Bo Liu arxiv

Hierarchical Reinforcement Learning (HRL) enhances the scalability of decision-making in long-horizon tasks by introducing temporal abstraction through options-policies that span multiple timesteps. Despite its theoretic…

Hierarchical Reinforcement Learning Change Point Detection