QFree: A Universal Value Function Factorization for Multi-Agent Reinforcement Learning

2023-11-01 · Rizhong Wang, Huiping Li, Di Cui, Demin Xu

Centralized training is widely utilized in the field of multi-agent reinforcement learning (MARL) to assure the stability of training process. Once a joint policy is obtained, it is critical to design a value function factorization method to extract optimal decentralized policies for the agents, which needs to satisfy the individual-global-max (IGM) principle. While imposing additional limitations on the IGM function class can help to meet the requirement, it comes at the cost of restricting its application to more complex multi-agent environments. In this paper, we propose QFree, a universal value function factorization method for MARL. We start by developing mathematical equivalent conditions of the IGM principle based on the advantage function, which ensures that the principle holds without any compromise, removing the conservatism of conventional methods. We then establish a more expressive mixing network architecture that can fulfill the equivalent factorization. In particular, the novel loss function is developed by considering the equivalent conditions as regularization term during policy evaluation in the MARL algorithm. Finally, the effectiveness of the proposed method is verified in a nonmonotonic matrix game scenario. Moreover, we show that QFree achieves the state-of-the-art performance in a general-purpose complex MARL benchmark environment, Starcraft Multi-Agent Challenge (SMAC).

📄 PDF Abstract BibTeX arXiv:2311.00356

Code (0)

등록된 구현이 없습니다.

Tasks

Multi-agent Reinforcement Learning reinforcement-learning SMAC SMAC+Starcraft

DFAC Framework: Factorizing the Value Function via Quantile Mixture for Multi-Agent Distributional Q-Learning

2021-02-16 · Wei-Fang Sun, Cheng-Kuang Lee, Chun-Yi Lee

In fully cooperative multi-agent reinforcement learning (MARL) settings, the environments are highly stochastic due to the partial observability of each agent and the continuously changing policies of the other agents. T…

Multi-agent Reinforcement Learning Q-Learning SMAC SMAC++1

ReMIX: Regret Minimization for Monotonic Value Function Factorization in Multiagent Reinforcement Learning

2023-02-11 · Yongsheng Mei, Hanhan Zhou, Tian Lan

Value function factorization methods have become a dominant approach for cooperative multiagent reinforcement learning under a centralized training and decentralized execution paradigm. By factorizing the optimal joint a…

Decision Making reinforcement-learning Reinforcement Learning (RL)Starcraft

A Unified Framework for Factorizing Distributional Value Functions for Multi-Agent Reinforcement Learning

2023-06-04 · Wei-Fang Sun, Cheng-Kuang Lee, Simon See, Chun-Yi Lee

In fully cooperative multi-agent reinforcement learning (MARL) settings, environments are highly stochastic due to the partial observability of each agent and the continuously changing policies of other agents. To addres…

Multi-agent Reinforcement Learning reinforcement-learning SMAC Starcraft

VAST: Value Function Factorization with Variable Agent Sub-Teams

2021-12-01 · NeurIPS 2021 12 · Thomy Phan, Fabian Ritz, Lenz Belzner, Philipp Altmann 외

Value function factorization (VFF) is a popular approach to cooperative multi-agent reinforcement learning in order to learn local value functions from global rewards. However, state-of-the-art VFF is limited to a handfu…

Multi-agent Reinforcement Learning

NQMIX: Non-monotonic Value Function Factorization for Deep Multi-Agent Reinforcement Learning

2021-04-05 · Quanlin Chen

Multi-agent value-based approaches recently make great progress, especially value decomposition methods. However, there are still a lot of limitations in value function factorization. In VDN, the joint action-value funct…

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning (RL)SMAC+3