Landmark-Guided Subgoal Generation in Hierarchical Reinforcement Learning

2021-10-26 · NeurIPS 2021 12 · Junsu Kim, Younggyo Seo, Jinwoo Shin

Goal-conditioned hierarchical reinforcement learning (HRL) has shown promising results for solving complex and long-horizon RL tasks. However, the action space of high-level policy in the goal-conditioned HRL is often large, so it results in poor exploration, leading to inefficiency in training. In this paper, we present HIerarchical reinforcement learning Guided by Landmarks (HIGL), a novel framework for training a high-level policy with a reduced action space guided by landmarks, i.e., promising states to explore. The key component of HIGL is twofold: (a) sampling landmarks that are informative for exploration and (b) encouraging the high-level policy to generate a subgoal towards a selected landmark. For (a), we consider two criteria: coverage of the entire visited state space (i.e., dispersion of states) and novelty of states (i.e., prediction error of a state). For (b), we select a landmark as the very first landmark in the shortest path in a graph whose nodes are landmarks. Our experiments demonstrate that our framework outperforms prior-arts across a variety of control tasks, thanks to efficient exploration guided by landmarks.

📄 PDF Abstract BibTeX arXiv:2110.13625

Code (1)

junsu-kim97/higl 공식 구현 pytorch

Tasks

Efficient ExplorationHierarchical Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Balancing Exploration and Exploitation in Hierarchical Reinforcement Learning via Latent Landmark Graphs

2023-07-22 · Qingyang Zhang, Yiming Yang, Jingqing Ruan, Xuantang Xiong 외

Goal-Conditioned Hierarchical Reinforcement Learning (GCHRL) is a promising paradigm to address the exploration-exploitation dilemma in reinforcement learning. It decomposes the source task into subgoal conditional subta…

continuous-control Continuous Control Hierarchical Reinforcement Learning reinforcement-learning+2

Solving Sokoban using Hierarchical Reinforcement Learning with Landmarks

2025-04-06 · Sergey Pastukhov

We introduce a novel hierarchical reinforcement learning (HRL) framework that performs top-down recursive planning via learned subgoals, successfully applied to the complex combinatorial puzzle game Sokoban. Our approach…

Hierarchical Reinforcement Learning reinforcement-learning Reinforcement Learning Sokoban

Option Discovery Using LLM-guided Semantic Hierarchical Reinforcement Learning

2025-03-24 · Chak Lam Shek, Pratap Tokekar

Large Language Models (LLMs) have shown remarkable promise in reasoning and decision-making, yet their integration with Reinforcement Learning (RL) for complex robotic tasks remains underexplored. In this paper, we propo…

Decision Making Hierarchical Reinforcement Learning reinforcement-learning Reinforcement Learning+1

Learning and Exploiting Multiple Subgoals for Fast Exploration in Hierarchical Reinforcement Learning

2019-05-13 · Libo Xing

Hierarchical Reinforcement Learning (HRL) exploits temporally extended actions, or options, to make decisions from a higher-dimensional perspective to alleviate the sparse reward problem, one of the most challenging prob…

Hierarchical Reinforcement Learning Montezuma's Revenge reinforcement-learning Reinforcement Learning+1

STO-RL: Offline RL under Sparse Rewards via LLM-Guided Subgoal Temporal Order

2026-01-13 · Chengyang Gu, Yuxin Pan, Hui Xiong, Yize Chen arxiv

Offline reinforcement learning (RL) enables policy learning from pre-collected datasets, avoiding costly and risky online interactions, but it often struggles with long-horizon tasks involving sparse rewards. Existing go…

Reinforcement Learning Offline RL