DHP: Discrete Hierarchical Planning for Hierarchical Reinforcement Learning Agents

2025-02-04 · Shashank Sharma, Janina Hoffmann, Vinay Namboodiri

Hierarchical Reinforcement Learning (HRL) agents often struggle with long-horizon visual planning due to their reliance on error-prone distance metrics. We propose Discrete Hierarchical Planning (DHP), a method that replaces continuous distance estimates with discrete reachability checks to evaluate subgoal feasibility. DHP recursively constructs tree-structured plans by decomposing long-term goals into sequences of simpler subtasks, using a novel advantage estimation strategy that inherently rewards shorter plans and generalizes beyond training depths. In addition, to address the data efficiency challenge, we introduce an exploration strategy that generates targeted training examples for the planning modules without needing expert data. Experiments in 25-room navigation environments demonstrate $100\%$ success rate (vs $82\%$ baseline) and $73$-step average episode length (vs $158$-step baseline). The method also generalizes to momentum-based control tasks and requires only $\log N$ steps for replanning. Theoretical analysis and ablations validate our design choices.

📄 PDF Abstract BibTeX arXiv:2502.01956

Code (0)

등록된 구현이 없습니다.

Tasks

Hierarchical Reinforcement Learning reinforcement-learning Reinforcement Learning

Learning Temporally Extended Skills in Continuous Domains as Symbolic Actions for Planning

2022-07-11 · Jan Achterhold, Markus Krimmel, Joerg Stueckler

Problems which require both long-horizon planning and continuous control capabilities pose significant challenges to existing reinforcement learning agents. In this paper we introduce a novel hierarchical reinforcement l…

continuous-control Continuous Control Hierarchical Reinforcement Learning reinforcement-learning+2

Hierarchical Reinforcement Learning in Complex 3D Environments

2023-02-28 · Bernardo Avila Pires, Feryal Behbahani, Hubert Soyer, Kyriacos Nikiforou 외

Hierarchical Reinforcement Learning (HRL) agents have the potential to demonstrate appealing capabilities such as planning and exploration with abstraction, transfer, and skill reuse. Recent successes with HRL across dif…

Deep Reinforcement Learning Hierarchical Reinforcement Learning reinforcement-learning Reinforcement Learning+1

Hierarchical Message-Passing Policies for Multi-Agent Reinforcement Learning

2025-07-31 · Tommaso Marzi, Cesare Alippi, Andrea Cini arxiv

Decentralized Multi-Agent Reinforcement Learning (MARL) methods allow for learning scalable multi-agent policies, but suffer from partial observability and induced non-stationarity. These challenges can be addressed by i…

Hierarchical Reinforcement Learning Multi-agent Reinforcement Learning

Dynamic planning in hierarchical active inference

2024-02-18 · Matteo Priorelli, Ivilin Peev Stoianov

By dynamic planning, we refer to the ability of the human brain to infer and impose motor trajectories related to cognitive decisions. A recent paradigm, active inference, brings fundamental insights into the adaptation …

Decision Making

PEORL: Integrating Symbolic Planning and Hierarchical Reinforcement Learning for Robust Decision-Making

2018-04-20 · Fangkai Yang, Daoming Lyu, Bo Liu, Steven Gustafson

Reinforcement learning and symbolic planning have both been used to build intelligent autonomous agents. Reinforcement learning relies on learning from interactions with real world, which often requires an unfeasibly lar…

Decision Making Hierarchical Reinforcement Learning reinforcement-learning Reinforcement Learning+1