On Improving Model-Free Algorithms for Decentralized Multi-Agent Reinforcement Learning

2021-10-12 · Weichao Mao, Lin F. Yang, Kaiqing Zhang, Tamer Başar

Multi-agent reinforcement learning (MARL) algorithms often suffer from an exponential sample complexity dependence on the number of agents, a phenomenon known as \emph{the curse of multiagents}. In this paper, we address this challenge by investigating sample-efficient model-free algorithms in \emph{decentralized} MARL, and aim to improve existing algorithms along this line. For learning (coarse) correlated equilibria in general-sum Markov games, we propose \emph{stage-based} V-learning algorithms that significantly simplify the algorithmic design and analysis of recent works, and circumvent a rather complicated no-\emph{weighted}-regret bandit subroutine. For learning Nash equilibria in Markov potential games, we propose an independent policy gradient algorithm with a decentralized momentum-based variance reduction technique. All our algorithms are decentralized in that each agent can make decisions based on only its local information. Neither communication nor centralized coordination is required during learning, leading to a natural generalization to a large number of agents. We also provide numerical simulations to corroborate our theoretical findings.

📄 PDF Abstract BibTeX arXiv:2110.05707

Code (0)

등록된 구현이 없습니다.

Tasks

Multi-agent Reinforcement LearningQ-Learningreinforcement-learning Reinforcement Learning (RL)Robot Navigation

Methods 이 논문이 사용한 방법론

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

Structured Cooperative Multi-Agent Reinforcement Learning: a Bayesian Network Perspective

2025-10-11 · Shahbaz P Qadri Syed, He Bai arxiv

The empirical success of multi-agent reinforcement learning (MARL) has motivated the search for more efficient and scalable algorithms for large scale multi-agent systems. However, existing state-of-the-art algorithms do…

Multi-agent Reinforcement Learning

MAMBPO: Sample-efficient multi-robot reinforcement learning using learned world models

2021-03-05 · Daniël Willemsen, Mario Coppola, Guido C. H. E. de Croon

Multi-robot systems can benefit from reinforcement learning (RL) algorithms that learn behaviours in a small number of trials, a property known as sample efficiency. This research thus investigates the use of learned wor…

reinforcement-learning Reinforcement Learning (RL)

Consensus Learning for Cooperative Multi-Agent Reinforcement Learning

2022-06-06 · Zhiwei Xu, Bin Zhang, Dapeng Li, Zeren Zhang 외

Almost all multi-agent reinforcement learning algorithms without communication follow the principle of centralized training with decentralized execution. During centralized training, agents can be guided by the same sign…

Contrastive Learning Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning+1

Decentralized Parameter-Free Online Learning

2025-10-17 · Tomas Ortega, Hamid Jafarkhani arxiv

We propose the first parameter-free decentralized online learning algorithms with network regret guarantees, which achieve sublinear regret without requiring hyperparameter tuning. This family of algorithms connects mult…

Fully Decentralized Cooperative Multi-Agent Reinforcement Learning: A Survey

2024-01-10 · Jiechuan Jiang, Kefan Su, Zongqing Lu

Cooperative multi-agent reinforcement learning is a powerful tool to solve many real-world cooperative tasks, but restrictions of real-world applications may require training the agents in a fully decentralized manner. D…

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning Survey