On Stateful Value Factorization in Multi-Agent Reinforcement Learning

2024-08-27 · Enrico Marchesini, Andrea Baisero, Rupali Bhati, Christopher Amato

Value factorization is a popular paradigm for designing scalable multi-agent reinforcement learning algorithms. However, current factorization methods make choices without full justification that may limit their performance. For example, the theory in prior work uses stateless (i.e., history) functions, while the practical implementations use state information -- making the motivating theory a mismatch for the implementation. Also, methods have built off of previous approaches, inheriting their architectures without exploring other, potentially better ones. To address these concerns, we formally analyze the theory of using the state instead of the history in current methods -- reconnecting theory and practice. We then introduce DuelMIX, a factorization algorithm that learns distinct per-agent utility estimators to improve performance and achieve full expressiveness. Experiments on StarCraft II micromanagement and Box Pushing tasks demonstrate the benefits of our intuitions.

📄 PDF Abstract BibTeX arXiv:2408.15381

Code (0)

등록된 구현이 없습니다.

Tasks

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement LearningStarcraftStarcraft II

A Unified Framework for Factorizing Distributional Value Functions for Multi-Agent Reinforcement Learning

2023-06-04 · Wei-Fang Sun, Cheng-Kuang Lee, Simon See, Chun-Yi Lee

In fully cooperative multi-agent reinforcement learning (MARL) settings, environments are highly stochastic due to the partial observability of each agent and the continuously changing policies of other agents. To addres…

Multi-agent Reinforcement Learning reinforcement-learning SMAC Starcraft

POWQMIX: Weighted Value Factorization with Potentially Optimal Joint Actions Recognition for Cooperative Multi-Agent Reinforcement Learning

2024-05-13 · Chang Huang, Shatong Zhu, Junqiao Zhao, Hongtu Zhou 외

Value function factorization methods are commonly used in cooperative multi-agent reinforcement learning, with QMIX receiving significant attention. Many QMIX-based methods introduce monotonicity constraints between the …

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning Starcraft+1

Offline Multi-Agent Reinforcement Learning with Coupled Value Factorization

2023-06-15 · Xiangsen Wang, Xianyuan Zhan

Offline reinforcement learning (RL) that learns policies from offline datasets without environment interaction has received considerable attention in recent years. Compared with the rich literature in the single-agent ca…

Management Multi-agent Reinforcement Learning Offline RL reinforcement-learning+4

FSV: Learning to Factorize Soft Value Function for Cooperative Multi-Agent Reinforcement Learning

2021-01-01 · Yueheng Li, Tianhao Zhang, Chen Wang, Jinan Sun 외

We explore energy-based solutions for cooperative multi-agent reinforcement learning (MARL) using the idea of function factorization in centralized training with decentralized execution (CTDE). Existing CTDE based factor…

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+2

DFAC Framework: Factorizing the Value Function via Quantile Mixture for Multi-Agent Distributional Q-Learning

2021-02-16 · Wei-Fang Sun, Cheng-Kuang Lee, Chun-Yi Lee

In fully cooperative multi-agent reinforcement learning (MARL) settings, the environments are highly stochastic due to the partial observability of each agent and the continuously changing policies of the other agents. T…

Multi-agent Reinforcement Learning Q-Learning SMAC SMAC++1