Learning and Exploiting Multiple Subgoals for Fast Exploration in Hierarchical Reinforcement Learning
Hierarchical Reinforcement Learning (HRL) exploits temporally extended actions, or options, to make decisions from a higher-dimensional perspective to alleviate the sparse reward problem, one of the most challenging problems in reinforcement learning. The majority of existing HRL algorithms require either significant manual design with respect to the specific environment or enormous exploration to automatically learn options from data. To achieve fast exploration without using manual design, we devise a multi-goal HRL algorithm, consisting of a high-level policy Manager and a low-level policy Worker. The Manager provides the Worker multiple subgoals at each time step. Each subgoal corresponds to an option to control the environment. Although the agent may show some confusion at the beginning of training since it is guided by three diverse subgoals, the agent's behavior policy will quickly learn how to respond to multiple subgoals from the high-level controller on different occasions. By exploiting multiple subgoals, the exploration efficiency is significantly improved. We conduct experiments in Atari's Montezuma's Revenge environment, a well-known sparse reward environment, and in doing so achieve the same performance as state-of-the-art HRL methods with substantially reduced training time cost.
Code (0)
등록된 구현이 없습니다.
Tasks
Hierarchical Reinforcement LearningMontezuma's Revengereinforcement-learningReinforcement LearningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Causality-driven Hierarchical Structure Discovery for Reinforcement Learning
Hierarchical reinforcement learning (HRL) effectively improves agents' exploration efficiency on tasks with sparse reward, with the guide of high-quality hierarchical structures (e.g., subgoals or options). However, how …
Hierarchical Reinforcement LearningMinecraftreinforcement-learningReinforcement Learning+1Active Hierarchical Exploration with Stable Subgoal Representation Learning
Goal-conditioned hierarchical reinforcement learning (GCHRL) provides a promising approach to solving long-horizon tasks. Recently, its success has been extended to more general settings by concurrently learning hierarch…
continuous-controlContinuous ControlHierarchical Reinforcement LearningRepresentation LearningMENTOR: Guiding Hierarchical Reinforcement Learning with Human Feedback and Dynamic Distance Constraint
Hierarchical reinforcement learning (HRL) provides a promising solution for complex tasks with sparse rewards of intelligent agents, which uses a hierarchical framework that divides tasks into subgoals and completes them…
Hierarchical Reinforcement Learningreinforcement-learningReinforcement LearningLearning Subgoal Representations with Slow Dynamics
In goal-conditioned Hierarchical Reinforcement Learning (HRL), a high-level policy periodically sets subgoals for a low-level policy, and the low-level policy is trained to reach those subgoals. A proper subgoal represen…
continuous-controlContinuous ControlHierarchical Reinforcement LearningRepresentation LearningFrom Words to Actions: Unveiling the Theoretical Underpinnings of LLM-Driven Autonomous Systems
In this work, from a theoretical lens, we aim to understand why large language model (LLM) empowered agents are able to solve decision-making problems in the physical world. To this end, consider a hierarchical reinforce…
Decision MakingHierarchical Reinforcement LearningImitation LearningIn-Context Learning+4