Contrasting Centralized and Decentralized Critics in Multi-Agent Reinforcement Learning

2021-02-08 · Xueguang Lyu, Yuchen Xiao, Brett Daley, Christopher Amato

Centralized Training for Decentralized Execution, where agents are trained offline using centralized information but execute in a decentralized manner online, has gained popularity in the multi-agent reinforcement learning community. In particular, actor-critic methods with a centralized critic and decentralized actors are a common instance of this idea. However, the implications of using a centralized critic in this context are not fully discussed and understood even though it is the standard choice of many algorithms. We therefore formally analyze centralized and decentralized critic approaches, providing a deeper understanding of the implications of critic choice. Because our theory makes unrealistic assumptions, we also empirically compare the centralized and decentralized critic methods over a wide set of environments to validate our theories and to provide practical advice. We show that there exist misconceptions regarding centralized critics in the current literature and show that the centralized critic design is not strictly beneficial, but rather both centralized and decentralized critics have different pros and cons that should be taken into account by algorithm designers.

📄 PDF Abstract BibTeX arXiv:2102.04402

Code (0)

등록된 구현이 없습니다.

Tasks

Misconceptions Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

On Centralized Critics in Multi-Agent Reinforcement Learning

2024-08-26 · Xueguang Lyu, Andrea Baisero, Yuchen Xiao, Brett Daley 외

Centralized Training for Decentralized Execution where agents are trained offline in a centralized fashion and execute online in a decentralized manner, has become a popular approach in Multi-Agent Reinforcement Learning…

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning Representation Learning

A Deeper Understanding of State-Based Critics in Multi-Agent Reinforcement Learning

2022-01-03 · Xueguang Lyu, Andrea Baisero, Yuchen Xiao, Christopher Amato

Centralized Training for Decentralized Execution, where training is done in a centralized offline fashion, has become a popular solution paradigm in Multi-Agent Reinforcement Learning. Many such methods take the form of …

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning (RL)

ACPO: Agent-Chained Policy Optimization for Multi-Agent Reinforcement Learning

2026-06-29 · Daiki E. Matsunaga, Junho Na, Tri Wahyu Guntara, Scott Sanner 외 arxiv

Cooperative tasks in Multi-Agent Reinforcement Learning (MARL) require agents to collectively maximize a shared return. Under the Centralized Training with Decentralized Execution (CTDE) paradigm, policy gradients have r…

Multi-agent Reinforcement Learning

Reducing Overestimation Bias in Multi-Agent Domains Using Double Centralized Critics

2019-10-03 · Johannes Ackermann, Volker Gabler, Takayuki Osa, Masashi Sugiyama

Many real world tasks require multiple agents to work together. Multi-agent reinforcement learning (RL) methods have been proposed in recent years to solve these tasks, but current methods often fail to efficiently learn…

Multi-agent Reinforcement Learning Reinforcement Learning Reinforcement Learning (RL)

TAPE: Leveraging Agent Topology for Cooperative Multi-Agent Policy Gradient

2023-12-25 · Xingzhou Lou, Junge Zhang, Timothy J. Norman, Kaiqi Huang 외

Multi-Agent Policy Gradient (MAPG) has made significant progress in recent years. However, centralized critics in state-of-the-art MAPG methods still face the centralized-decentralized mismatch (CDM) issue, which means s…