Revisiting the Monotonicity Constraint in Cooperative Multi-Agent Reinforcement Learning

2021-09-29 · Jian Hu, Siyang Jiang, Seth Austin Harding, Haibin Wu, Shih-wei Liao

QMIX, a popular MARL algorithm based on the monotonicity constraint, has been used as a baseline for the benchmark environments, such as Starcraft Multi-Agent Challenge (SMAC), Predator-Prey (PP). Recent variants of QMIX target relaxing the monotonicity constraint of QMIX to improve the expressive power of QMIX, allowing for performance improvement in SMAC. However, we find that such performance improvements of the variants are significantly affected by various implementation tricks. In this paper, we revisit the monotonicity constraint of QMIX, (1) we design a novel model RMC to further investigate the monotonicity constraint; the results show that monotonicity constraint can improve sample efficiency in some purely cooperative tasks; (2) we then re-evaluate the performance of QMIX and these variants by a grid hyperparameter search for the tricks; the results show QMIX achieves the best performance among them, achieving SOTA performance on SMAC and PP; (3) we analyze the monotonic mixing network from a theoretical perspective and show that it can represent any tasks which can be interpreted as purely cooperative. These analyses demonstrate that relaxing the monotonicity constraint of the mixing network will not always improve the performance of QMIX, which breaks our previous impressions of the monotonicity constraints.

📄 PDF Abstract BibTeX

Code (0)

등록된 구현이 없습니다.

Tasks

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)SMAC SMAC+Starcraft

Rethinking the Implementation Matters in Cooperative Multi-Agent Reinforcement Learning

2021-02-06 · Jian Hu, Siyang Jiang, Seth Austin Harding, Haibin Wu 외

Multi-Agent Reinforcement Learning (MARL) has seen revolutionary breakthroughs with its successful application to multi-agent cooperative tasks such as computer games and robot swarms. QMIX, a widely popular MARL algorit…

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+4

QR-MIX: Distributional Value Function Factorisation for Cooperative Multi-Agent Reinforcement Learning

2020-09-09 · Jian Hu, Seth Austin Harding, Haibin Wu, Siyue Hu 외

In Cooperative Multi-Agent Reinforcement Learning (MARL) and under the setting of Centralized Training with Decentralized Execution (CTDE), agents observe and interact with their environment locally and independently. Wi…

Multi-agent Reinforcement Learning quantile regression reinforcement-learning Reinforcement Learning (RL)+3

POWQMIX: Weighted Value Factorization with Potentially Optimal Joint Actions Recognition for Cooperative Multi-Agent Reinforcement Learning

2024-05-13 · Chang Huang, Shatong Zhu, Junqiao Zhao, Hongtu Zhou 외

Value function factorization methods are commonly used in cooperative multi-agent reinforcement learning, with QMIX receiving significant attention. Many QMIX-based methods introduce monotonicity constraints between the …

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning Starcraft+1

Beyond Monotonicity: Revisiting Factorization Principles in Multi-Agent Q-Learning

2025-11-12 · Tianmeng Hu, Yongzheng Cui, Rui Tang, Biao Luo 외 arxiv

Value decomposition is a central approach in multi-agent reinforcement learning (MARL), enabling centralized training with decentralized execution by factorizing the global value function into local values. To ensure ind…

Multi-agent Reinforcement Learning

QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning

2019-05-14 · Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero 외

We explore value-based solutions for multi-agent reinforcement learning (MARL) tasks in the centralized training with decentralized execution (CTDE) regime popularized recently. However, VDN and QMIX are representative e…

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1