SMIX($λ$): Enhancing Centralized Value Functions for Cooperative Multi-Agent Reinforcement Learning

2019-11-11 · Xinghu Yao, Chao Wen, Yuhui Wang, Xiaoyang Tan

Learning a stable and generalizable centralized value function (CVF) is a crucial but challenging task in multi-agent reinforcement learning (MARL), as it has to deal with the issue that the joint action space increases exponentially with the number of agents in such scenarios. This paper proposes an approach, named SMIX(${\lambda}$), to address the issue using an efficient off-policy centralized training method within a flexible learner search space. As importance sampling for such off-policy training is both computationally costly and numerically unstable, we proposed to use the ${\lambda}$-return as a proxy to compute the TD error. With this new loss function objective, we adopt a modified QMIX network structure as the base to train our model. By further connecting it with the ${Q(\lambda)}$ approach from an unified expectation correction viewpoint, we show that the proposed SMIX(${\lambda}$) is equivalent to ${Q(\lambda)}$ and hence shares its convergence properties, while without being suffered from the aforementioned curse of dimensionality problem inherent in MARL. Experiments on the StarCraft Multi-Agent Challenge (SMAC) benchmark demonstrate that our approach not only outperforms several state-of-the-art MARL methods by a large margin, but also can be used as a general tool to improve the overall performance of other CTDE-type algorithms by enhancing their CVFs.

📄 PDF Abstract BibTeX arXiv:1911.04094

Code (1)

chaovven/SMIX 공식 구현 pytorch

Tasks

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)SMAC SMAC+Starcraft

Transformer-based Value Function Decomposition for Cooperative Multi-agent Reinforcement Learning in StarCraft

2022-08-15 · Muhammad Junaid Khan, Syed Hammad Ahmed, Gita Sukthankar

The StarCraft II Multi-Agent Challenge (SMAC) was created to be a challenging benchmark problem for cooperative multi-agent reinforcement learning (MARL). SMAC focuses exclusively on the problem of StarCraft micromanagem…

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning (RL)SMAC+3

Centralized Cooperative Exploration Policy for Continuous Control Tasks

2023-01-06 · Chao Li, Chen Gong, Qiang He, Xinwen Hou 외

The deep reinforcement learning (DRL) algorithm works brilliantly on solving various complex control tasks. This phenomenal success can be partly attributed to DRL encouraging intelligent agents to sufficiently explore t…

continuous-control Continuous Control Deep Reinforcement Learning

Value-Decomposition Networks For Cooperative Multi-Agent Learning

2017-06-16 · Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki 외

We study the problem of cooperative multi-agent reinforcement learning with a single joint reward signal. This class of learning problems is difficult because of the often large combined action and observation spaces. In…

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1

Approximate Linear Programming for Decentralized Policy Iteration in Cooperative Multi-agent Markov Decision Processes

2023-11-20 · Lakshmi Mandal, Chandrashekar Lakshminarayanan, Shalabh Bhatnagar

In this work, we consider a cooperative multi-agent Markov decision process (MDP) involving m agents. At each decision epoch, all the m agents independently select actions in order to maximize a common long-term objectiv…

Automata-Conditioned Cooperative Multi-Agent Reinforcement Learning

2025-11-04 · Beyazit Yalcinkaya, Marcell Vazquez-Chanlatte, Ameesh Shah, Hanna Krasowski 외 arxiv

We study learning multi-task, multi-agent policies for cooperative, temporal objectives, under centralized training, decentralized execution. In this setting, using automata to represent tasks assigned to agents enables …

Multi-agent Reinforcement Learning