VAST: Value Function Factorization with Variable Agent Sub-Teams
Value function factorization (VFF) is a popular approach to cooperative multi-agent reinforcement learning in order to learn local value functions from global rewards. However, state-of-the-art VFF is limited to a handful of agents in most domains. We hypothesize that this is due to the flat factorization scheme, where the VFF operator becomes a performance bottleneck with an increasing number of agents. Therefore, we propose VFF with variable agent sub-teams (VAST). VAST approximates a factorization for sub-teams which can be defined in an arbitrary way and vary over time, e.g., to adapt to different situations. The sub-team values are then linearly decomposed for all sub-team members. Thus, VAST can learn on a more focused and compact input representation of the original VFF operator. We evaluate VAST in three multi-agent domains and show that VAST can significantly outperform state-of-the-art VFF, when the number of agents is sufficiently large.
Code (1)
Tasks
Multi-agent Reinforcement LearningSimilar Papers 제목 키워드 기반
DFAC Framework: Factorizing the Value Function via Quantile Mixture for Multi-Agent Distributional Q-Learning
In fully cooperative multi-agent reinforcement learning (MARL) settings, the environments are highly stochastic due to the partial observability of each agent and the continuously changing policies of the other agents. T…
Multi-agent Reinforcement LearningQ-LearningSMACSMAC++1A Unified Framework for Factorizing Distributional Value Functions for Multi-Agent Reinforcement Learning
In fully cooperative multi-agent reinforcement learning (MARL) settings, environments are highly stochastic due to the partial observability of each agent and the continuously changing policies of other agents. To addres…
Multi-agent Reinforcement Learningreinforcement-learningSMACStarcraftReMIX: Regret Minimization for Monotonic Value Function Factorization in Multiagent Reinforcement Learning
Value function factorization methods have become a dominant approach for cooperative multiagent reinforcement learning under a centralized training and decentralized execution paradigm. By factorizing the optimal joint a…
Decision Makingreinforcement-learningReinforcement Learning (RL)StarcraftNQMIX: Non-monotonic Value Function Factorization for Deep Multi-Agent Reinforcement Learning
Multi-agent value-based approaches recently make great progress, especially value decomposition methods. However, there are still a lot of limitations in value function factorization. In VDN, the joint action-value funct…
Multi-agent Reinforcement Learningreinforcement-learningReinforcement Learning (RL)SMAC+3Value Functions Factorization with Latent State Information Sharing in Decentralized Multi-Agent Policy Gradients
Value function factorization via centralized training and decentralized execution is promising for solving cooperative multi-agent reinforcement tasks. One of the approaches in this area, QMIX, has become state-of-the-ar…
StarcraftStarcraft IIVariational Inference