Heterogeneous-Agent Mirror Learning: A Continuum of Solutions to Cooperative MARL

2022-08-02 · Jakub Grudzien Kuba, Xidong Feng, Shiyao Ding, Hao Dong, Jun Wang, Yaodong Yang

The necessity for cooperation among intelligent machines has popularised cooperative multi-agent reinforcement learning (MARL) in the artificial intelligence (AI) research community. However, many research endeavors have been focused on developing practical MARL algorithms whose effectiveness has been studied only empirically, thereby lacking theoretical guarantees. As recent studies have revealed, MARL methods often achieve performance that is unstable in terms of reward monotonicity or suboptimal at convergence. To resolve these issues, in this paper, we introduce a novel framework named Heterogeneous-Agent Mirror Learning (HAML) that provides a general template for MARL algorithmic designs. We prove that algorithms derived from the HAML template satisfy the desired properties of the monotonic improvement of the joint reward and the convergence to Nash equilibrium. We verify the practicality of HAML by proving that the current state-of-the-art cooperative MARL algorithms, HATRPO and HAPPO, are in fact HAML instances. Next, as a natural outcome of our theory, we propose HAML extensions of two well-known RL algorithms, HAA2C (for A2C) and HADDPG (for DDPG), and demonstrate their effectiveness against strong baselines on StarCraftII and Multi-Agent MuJoCo tasks.

📄 PDF Abstract BibTeX arXiv:2208.01682

Code (0)

등록된 구현이 없습니다.

Tasks

MuJoCoMulti-agent Reinforcement Learning

Heterogeneous Multi-Agent Reinforcement Learning via Mirror Descent Policy Optimization

2023-08-13 · Mohammad Mehdi Nasiri, Mansoor Rezghi

This paper presents an extension of the Mirror Descent method to overcome challenges in cooperative Multi-Agent Reinforcement Learning (MARL) settings, where agents have varying abilities and individual policies. The pro…

LEMMA MuJoCo Multi-agent Reinforcement Learning reinforcement-learning+1

Heterogeneous-Agent Reinforcement Learning

2023-04-19 · Yifan Zhong, Jakub Grudzien Kuba, Xidong Feng, Siyi Hu 외

The necessity for cooperation among intelligent machines has popularised cooperative multi-agent reinforcement learning (MARL) in AI research. However, many research endeavours heavily rely on parameter sharing among age…

LEMMA Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning

Maximum Entropy Heterogeneous-Agent Reinforcement Learning

2023-06-19 · Jiarong Liu, Yifan Zhong, Siyi Hu, Haobo Fu 외

Multi-agent reinforcement learning (MARL) has been shown effective for cooperative games in recent years. However, existing state-of-the-art methods face challenges related to sample complexity, training instability, and…

MuJoCo Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning+1

Cooperative Policy Learning with Pre-trained Heterogeneous Observation Representations

2020-12-24 · Wenlei Shi, Xinran Wei, Jia Zhang, Xiaoyuan Ni 외

Multi-agent reinforcement learning (MARL) has been increasingly explored to learn the cooperative policy towards maximizing a certain global reward. Many existing studies take advantage of graph neural networks (GNN) in …

Decoder Graph Attention Multi-agent Reinforcement Learning

Deep Reinforcement Learning for Routing a Heterogeneous Fleet of Vehicles

2019-12-06 · Jose Manuel Vera, Andres G. Abad

Motivated by the promising advances of deep-reinforcement learning (DRL) applied to cooperative multi-agent systems we propose a model and learning procedure to solve the Capacitated Multi-Vehicle Routing Problem (CMVRP)…

Deep Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)