FSV: Learning to Factorize Soft Value Function for Cooperative Multi-Agent Reinforcement Learning

2021-01-01 · Yueheng Li, Tianhao Zhang, Chen Wang, Jinan Sun, Shikun Zhang, Guangming Xie

We explore energy-based solutions for cooperative multi-agent reinforcement learning (MARL) using the idea of function factorization in centralized training with decentralized execution (CTDE). Existing CTDE based factorization methods are susceptible to the relative overgeneralization, where finding a suboptimal Nash Equilibrium, which is a well-known game-theoretic pathology. To resolve this issue, we propose a novel factorization method for cooperative MARL, named FSV, which learns to factorize the joint soft value function into individual ones for decentralized execution. Theoretical analysis shows that FSV solves a rich class of factorization tasks. Our experiment for the well-known task of the Max of Two Quadratics game shows that FSV fully converges to global optima in the joint action space in the continuous tasks by local searching in the joint action space. We evaluate FSV on a challenging set of StarCraft II micromanagement tasks, and show that FSV significantly outperforms existing factorization multi-agent reinforcement learning methods.

📄 PDF Abstract BibTeX

Code (0)

등록된 구현이 없습니다.

Tasks

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)StarcraftStarcraft II

Inverse Factorized Q-Learning for Cooperative Multi-agent Imitation Learning

2023-10-10 · The Viet Bui, Tien Mai, Thanh Hong Nguyen

This paper concerns imitation learning (IL) (i.e, the problem of learning to mimic expert behaviors from demonstrations) in cooperative multi-agent systems. The learning problem under consideration poses several challeng…

Imitation Learning Q-Learning

Towards Understanding Cooperative Multi-Agent Q-Learning with Value Factorization

2020-05-31 · NeurIPS 2021 12 · Jianhao Wang, Zhizhou Ren, Beining Han, Jianing Ye 외

Value factorization is a popular and promising approach to scaling up multi-agent reinforcement learning in cooperative settings, which balances the learning scalability and the representational capacity of value functio…

counterfactual Multi-agent Reinforcement Learning Q-Learning Starcraft+1

A Unified Framework for Factorizing Distributional Value Functions for Multi-Agent Reinforcement Learning

2023-06-04 · Wei-Fang Sun, Cheng-Kuang Lee, Simon See, Chun-Yi Lee

In fully cooperative multi-agent reinforcement learning (MARL) settings, environments are highly stochastic due to the partial observability of each agent and the continuously changing policies of other agents. To addres…

Multi-agent Reinforcement Learning reinforcement-learning SMAC Starcraft

DFAC Framework: Factorizing the Value Function via Quantile Mixture for Multi-Agent Distributional Q-Learning

2021-02-16 · Wei-Fang Sun, Cheng-Kuang Lee, Chun-Yi Lee

In fully cooperative multi-agent reinforcement learning (MARL) settings, the environments are highly stochastic due to the partial observability of each agent and the continuously changing policies of the other agents. T…

Multi-agent Reinforcement Learning Q-Learning SMAC SMAC++1

QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning

2019-05-14 · Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero 외

We explore value-based solutions for multi-agent reinforcement learning (MARL) tasks in the centralized training with decentralized execution (CTDE) regime popularized recently. However, VDN and QMIX are representative e…

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1