Graph Convolutional Value Decomposition in Multi-Agent Reinforcement Learning
We propose a novel framework for value function factorization in multi-agent deep reinforcement learning (MARL) using graph neural networks (GNNs). In particular, we consider the team of agents as the set of nodes of a complete directed graph, whose edge weights are governed by an attention mechanism. Building upon this underlying graph, we introduce a mixing GNN module, which is responsible for i) factorizing the team state-action value function into individual per-agent observation-action value functions, and ii) explicit credit assignment to each agent in terms of fractions of the global team reward. Our approach, which we call GraphMIX, follows the centralized training and decentralized execution paradigm, enabling the agents to make their decisions independently once training is completed. We show the superiority of GraphMIX as compared to the state-of-the-art on several scenarios in the StarCraft II multi-agent challenge (SMAC) benchmark. We further demonstrate how GraphMIX can be used in conjunction with a recent hierarchical MARL architecture to both improve the agents' performance and enable fine-tuning them on mismatched test scenarios with higher numbers of agents and/or actions.
Code (1)
Tasks
Deep Reinforcement LearningMulti-agent Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)SMACSMAC+StarcraftStarcraft IISimilar Papers 제목 키워드 기반
Cooperative Multi-Agent Reinforcement Learning with Hypergraph Convolution
Recent years have witnessed the great success of multi-agent systems (MAS). Value decomposition, which decomposes joint action values into individual action values, has been an important work in MAS. However, many value …
Multi-agent Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)+4Non-Linear Coordination Graphs
Value decomposition multi-agent reinforcement learning methods learn the global value function as a mixing of each agent's individual utility functions. Coordination graphs (CGs) represent a higher-order decomposition by…
Multi-agent Reinforcement LearningWireless Communication Enhanced Value Decomposition for Multi-Agent Reinforcement Learning
Cooperation in multi-agent reinforcement learning (MARL) benefits from inter-agent communication, yet most approaches assume idealized channels and existing value decomposition methods ignore who successfully shared info…
Multi-agent Reinforcement LearningTowards Understanding Linear Value Decomposition in Cooperative Multi-Agent Q-Learning
Value decomposition is a popular and promising approach to scaling up multi-agent reinforcement learning in cooperative settings. However, the theoretical understanding of such methods is limited. In this paper, we intro…
counterfactualMulti-agent Reinforcement LearningQ-LearningStarcraft+1Value Function Decomposition for Iterative Design of Reinforcement Learning Agents
Designing reinforcement learning (RL) agents is typically a difficult process that requires numerous design iterations. Learning can fail for a multitude of reasons, and standard RL methods provide too few tools to provi…
Decision Makingreinforcement-learningReinforcement LearningReinforcement Learning (RL)