Active Hierarchical Exploration with Stable Subgoal Representation Learning

2021-05-31 · ICLR 2022 4 · Siyuan Li, Jin Zhang, Jianhao Wang, Yang Yu, Chongjie Zhang

Goal-conditioned hierarchical reinforcement learning (GCHRL) provides a promising approach to solving long-horizon tasks. Recently, its success has been extended to more general settings by concurrently learning hierarchical policies and subgoal representations. Although GCHRL possesses superior exploration ability by decomposing tasks via subgoals, existing GCHRL methods struggle in temporally extended tasks with sparse external rewards, since the high-level policy learning relies on external rewards. As the high-level policy selects subgoals in an online learned representation space, the dynamic change of the subgoal space severely hinders effective high-level exploration. In this paper, we propose a novel regularization that contributes to both stable and efficient subgoal representation learning. Building upon the stable representation, we design measures of novelty and potential for subgoals, and develop an active hierarchical exploration strategy that seeks out new promising subgoals and states without intrinsic rewards. Experimental results show that our approach significantly outperforms state-of-the-art baselines in continuous control tasks with sparse rewards.

📄 PDF Abstract BibTeX arXiv:2105.14750

Code (1)

siyuanlee/hess 공식 구현 pytorch

Tasks

continuous-control Continuous Control Hierarchical Reinforcement Learning Representation Learning

Balancing Exploration and Exploitation in Hierarchical Reinforcement Learning via Latent Landmark Graphs

2023-07-22 · Qingyang Zhang, Yiming Yang, Jingqing Ruan, Xuantang Xiong 외

Goal-Conditioned Hierarchical Reinforcement Learning (GCHRL) is a promising paradigm to address the exploration-exploitation dilemma in reinforcement learning. It decomposes the source task into subgoal conditional subta…

continuous-control Continuous Control Hierarchical Reinforcement Learning reinforcement-learning+2

Learning Subgoal Representations with Slow Dynamics

2021-01-01 · ICLR 2021 1 · Siyuan Li, Lulu Zheng, Jianhao Wang, Chongjie Zhang

In goal-conditioned Hierarchical Reinforcement Learning (HRL), a high-level policy periodically sets subgoals for a low-level policy, and the low-level policy is trained to reach those subgoals. A proper subgoal represen…

continuous-control Continuous Control Hierarchical Reinforcement Learning Representation Learning

Efficient Exploration through Intrinsic Motivation Learning for Unsupervised Subgoal Discovery in Model-Free Hierarchical Reinforcement Learning

2019-11-18 · Jacob Rafati, David C. Noelle

Efficient exploration for automatic subgoal discovery is a challenging problem in Hierarchical Reinforcement Learning (HRL). In this paper, we show that intrinsic motivation learning increases the efficiency of explorati…

Efficient Exploration Hierarchical Reinforcement Learning reinforcement-learning Reinforcement Learning+1

MENTOR: Guiding Hierarchical Reinforcement Learning with Human Feedback and Dynamic Distance Constraint

2024-02-22 · Xinglin Zhou, Yifu Yuan, Shaofu Yang, Jianye Hao

Hierarchical reinforcement learning (HRL) provides a promising solution for complex tasks with sparse rewards of intelligent agents, which uses a hierarchical framework that divides tasks into subgoals and completes them…

Hierarchical Reinforcement Learning reinforcement-learning Reinforcement Learning

Strict Subgoal Execution: Reliable Long-Horizon Planning in Hierarchical Reinforcement Learning

2025-06-26 · Jaebak Hwang, Sanghyeon Lee, Jeongmo Kim, Seungyul Han

Long-horizon goal-conditioned tasks pose fundamental challenges for reinforcement learning (RL), particularly when goals are distant and rewards are sparse. While hierarchical and graph-based methods offer partial soluti…

Decision Making Hierarchical Reinforcement Learning Reinforcement Learning (RL)