Heterogeneous-Agent Reinforcement Learning

2023-04-19 · Yifan Zhong, Jakub Grudzien Kuba, Xidong Feng, Siyi Hu, Jiaming Ji, Yaodong Yang

The necessity for cooperation among intelligent machines has popularised cooperative multi-agent reinforcement learning (MARL) in AI research. However, many research endeavours heavily rely on parameter sharing among agents, which confines them to only homogeneous-agent setting and leads to training instability and lack of convergence guarantees. To achieve effective cooperation in the general heterogeneous-agent setting, we propose Heterogeneous-Agent Reinforcement Learning (HARL) algorithms that resolve the aforementioned issues. Central to our findings are the multi-agent advantage decomposition lemma and the sequential update scheme. Based on these, we develop the provably correct Heterogeneous-Agent Trust Region Learning (HATRL), and derive HATRPO and HAPPO by tractable approximations. Furthermore, we discover a novel framework named Heterogeneous-Agent Mirror Learning (HAML), which strengthens theoretical guarantees for HATRPO and HAPPO and provides a general template for cooperative MARL algorithmic designs. We prove that all algorithms derived from HAML inherently enjoy monotonic improvement of joint return and convergence to Nash Equilibrium. As its natural outcome, HAML validates more novel algorithms in addition to HATRPO and HAPPO, including HAA2C, HADDPG, and HATD3, which generally outperform their existing MA-counterparts. We comprehensively test HARL algorithms on six challenging benchmarks and demonstrate their superior effectiveness and stability for coordinating heterogeneous agents compared to strong baselines such as MAPPO and QMIX.

📄 PDF Abstract BibTeX arXiv:2304.09870

Code (1)

pku-marl/harl 공식 구현 pytorch

Tasks

LEMMAMulti-agent Reinforcement Learning reinforcement-learning Reinforcement Learning

Methods 이 논문이 사용한 방법론

Test 설명 없음

Cooperative Heterogeneous Deep Reinforcement Learning

2020-11-02 · NeurIPS 2020 12 · Han Zheng, Pengfei Wei, Jing Jiang, Guodong Long 외

Numerous deep reinforcement learning agents have been proposed, and each of them has its strengths and flaws. In this work, we present a Cooperative Heterogeneous Deep Reinforcement Learning (CHDRL) framework that can le…

continuous-control Continuous Control Deep Reinforcement Learning Evolutionary Algorithms+4

Prioritized League Reinforcement Learning for Large-Scale Heterogeneous Multiagent Systems

2024-03-26 · Qingxu Fu, Zhiqiang Pu, Min Chen, Tenghai Qiu 외

Large-scale heterogeneous multiagent systems feature various realistic factors in the real world, such as agents with diverse abilities and overall system cost. In comparison to homogeneous systems, heterogeneous systems…

reinforcement-learning Reinforcement Learning

Heterogeneous Agent Collaborative Reinforcement Learning

2026-03-03 · Zhixia Zhang, Zixuan Huang, Gongxun Li, Huaiyang Wang 외 arxiv

We introduce Heterogeneous Agent Collaborative Reinforcement Learning (HACRL), a new Reinforcement Learning from Verifiable Reward (RLVR) problem that addresses the inefficiencies of isolated multi-agent on-policy optimi…

Multi-agent Reinforcement Learning

Learning Heterogeneous Agent Cooperation via Multiagent League Training

2022-11-13 · Qingxu Fu, Xiaolin Ai, Jianqiang Yi, Tenghai Qiu 외

Many multiagent systems in the real world include multiple types of agents with different abilities and functionality. Such heterogeneous multiagent systems have significant practical advantages. However, they also come …

Diversity reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

CH-MARL: A Multimodal Benchmark for Cooperative, Heterogeneous Multi-Agent Reinforcement Learning

2022-08-26 · Vasu Sharma, Prasoon Goyal, Kaixiang Lin, Govind Thattai 외

We propose a multimodal (vision-and-language) benchmark for cooperative and heterogeneous multi-agent learning. We introduce a benchmark multimodal dataset with tasks involving collaboration between multiple simulated he…

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)