Inter-Level Cooperation in Hierarchical Reinforcement Learning

2019-12-05 · Abdul Rahman Kreidieh, Glen Berseth, Brandon Trabucco, Samyak Parajuli, Sergey Levine, Alexandre M. Bayen

Hierarchies of temporally decoupled policies present a promising approach for enabling structured exploration in complex long-term planning problems. To fully achieve this approach an end-to-end training paradigm is needed. However, training these multi-level policies has had limited success due to challenges arising from interactions between the goal-assigning and goal-achieving levels within a hierarchy. In this article, we consider the policy optimization process as a multi-agent process. This allows us to draw on connections between communication and cooperation in multi-agent RL, and demonstrate the benefits of increased cooperation between sub-policies on the training performance of the overall policy. We introduce a simple yet effective technique for inducing inter-level cooperation by modifying the objective function and subsequent gradients of higher-level policies. Experimental results on a wide variety of simulated robotics and traffic control tasks demonstrate that inducing cooperation results in stronger performing policies and increased sample efficiency on a set of difficult long time horizon tasks. We also find that goal-conditioned policies trained using our method display better transfer to new tasks, highlighting the benefits of our method in learning task-agnostic lower-level behaviors. Videos and code are available at: https://sites.google.com/berkeley.edu/cooperative-hrl.

📄 PDF Abstract BibTeX arXiv:1912.02368

Code (1)

AboudyKreidieh/h-baselines 공식 구현 tf

Tasks

Hierarchical Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Hierarchical Reinforcement Learning with Opponent Modeling for Distributed Multi-agent Cooperation

2022-06-25 · Zhixuan Liang, Jiannong Cao, Shan Jiang, Divya Saxena 외

Many real-world applications can be formulated as multi-agent cooperation problems, such as network packet routing and coordination of autonomous vehicles. The emergence of deep reinforcement learning (DRL) provides a pr…

Autonomous Vehicles Decision Making Deep Reinforcement Learning Hierarchical Reinforcement Learning+3

Guided Cooperation in Hierarchical Reinforcement Learning via Model-based Rollout

2023-09-24 · Haoran Wang, Zeshen Tang, Leya Yang, Yaoru Sun 외

Goal-conditioned hierarchical reinforcement learning (HRL) presents a promising approach for enabling effective exploration in complex, long-horizon reinforcement learning (RL) tasks through temporal abstraction. Empiric…

Hierarchical Reinforcement Learning reinforcement-learning Reinforcement Learning (RL)

VIKI-R: Coordinating Embodied Multi-Agent Cooperation via Reinforcement Learning

2025-06-10 · Li Kang, Xiufeng Song, Heng Zhou, Yiran Qin 외

Coordinating multiple embodied agents in dynamic environments remains a core challenge in artificial intelligence, requiring both perception-driven reasoning and scalable cooperation strategies. While recent works have l…

Task Planning Visual Reasoning

Hierarchical Reinforcement Learning for Optimal Agent Grouping in Cooperative Systems

2025-01-11 · Liyuan Hu

This paper presents a hierarchical reinforcement learning (RL) approach to address the agent grouping or pairing problem in cooperative multi-agent systems. The goal is to simultaneously learn the optimal grouping and ag…

Decision Making Hierarchical Reinforcement Learning reinforcement-learning Reinforcement Learning+1

Self-Clustering Hierarchical Multi-Agent Reinforcement Learning with Extensible Cooperation Graph

2024-03-26 · Qingxu Fu, Tenghai Qiu, Jianqiang Yi, Zhiqiang Pu 외

Multi-Agent Reinforcement Learning (MARL) has been successful in solving many cooperative challenges. However, classic non-hierarchical MARL algorithms still cannot address various complex multi-agent problems that requi…

Clustering Graph Learning Multi-agent Reinforcement Learning