Off-Policy Multi-Agent Decomposed Policy Gradients

2020-07-24 · Yihan Wang, Beining Han, Tonghan Wang, Heng Dong, Chongjie Zhang

Multi-agent policy gradient (MAPG) methods recently witness vigorous progress. However, there is a significant performance discrepancy between MAPG methods and state-of-the-art multi-agent value-based approaches. In this paper, we investigate causes that hinder the performance of MAPG algorithms and present a multi-agent decomposed policy gradient method (DOP). This method introduces the idea of value function decomposition into the multi-agent actor-critic framework. Based on this idea, DOP supports efficient off-policy learning and addresses the issue of centralized-decentralized mismatch and credit assignment in both discrete and continuous action spaces. We formally show that DOP critics have sufficient representational capability to guarantee convergence. In addition, empirical evaluations on the StarCraft II micromanagement benchmark and multi-agent particle environments demonstrate that DOP significantly outperforms both state-of-the-art value-based and policy-based multi-agent reinforcement learning algorithms. Demonstrative videos are available at https://sites.google.com/view/dop-mapg/.

📄 PDF Abstract BibTeX arXiv:2007.12322

Code (1)

TonghanWang/DOP pytorch

Tasks

Multi-agent Reinforcement LearningStarcraftStarcraft II

DOP: Off-Policy Multi-Agent Decomposed Policy Gradients

2021-01-01 · ICLR 2021 1 · Yihan Wang, Beining Han, Tonghan Wang, Heng Dong 외

Multi-agent Reinforcement Learning Starcraft Starcraft II

Decomposed Soft Actor-Critic Method for Cooperative Multi-Agent Reinforcement Learning

2021-04-14 · Yuan Pu, Shaochen Wang, Rui Yang, Xin Yao 외

Deep reinforcement learning methods have shown great performance on many challenging cooperative multi-agent tasks. Two main promising research directions are multi-agent value function decomposition and multi-agent poli…

counterfactual Deep Reinforcement Learning Multi-agent Reinforcement Learning reinforcement-learning+4

Policy Diagnosis via Measuring Role Diversity in Cooperative Multi-agent RL

2022-06-01 · Siyi Hu, Chuanlong Xie, Xiaodan Liang, Xiaojun Chang

Cooperative multi-agent reinforcement learning (MARL) is making rapid progress for solving tasks in a grid world and real-world scenarios, in which agents are given different attributes and goals, resulting in different …

Diversity Multi-agent Reinforcement Learning SMAC SMAC++1

Almost Sure Convergence of Networked Policy Gradient over Time-Varying Networks in Markov Potential Games

2024-10-26 · Sarper Aydin, Ceyhun Eksin

We propose networked policy gradient play for solving Markov potential games including continuous action and state spaces. In the decentralized algorithm, agents sample their actions from parametrized and differentiable …

Difference Rewards Policy Gradients

2020-12-21 · Jacopo Castellini, Sam Devlin, Frans A. Oliehoek, Rahul Savani

Policy gradient methods have become one of the most popular classes of algorithms for multi-agent reinforcement learning. A key challenge, however, that is not addressed by many of these methods is multi-agent credit ass…

counterfactual Multi-agent Reinforcement Learning Policy Gradient Methods Reinforcement Learning (RL)