AgentMixer: Multi-Agent Correlated Policy Factorization
In multi-agent reinforcement learning, centralized training with decentralized execution (CTDE) methods typically assume that agents make decisions based on their local observations independently, which may not lead to a correlated joint policy with coordination. Coordination can be explicitly encouraged during training and individual policies can be trained to imitate the correlated joint policy. However, this may lead to an \textit{asymmetric learning failure} due to the observation mismatch between the joint and individual policies. Inspired by the concept of correlated equilibrium, we introduce a \textit{strategy modification} called AgentMixer that allows agents to correlate their policies. AgentMixer combines individual partially observable policies into a joint fully observable policy non-linearly. To enable decentralized execution, we introduce \textit{Individual-Global-Consistency} to guarantee mode consistency during joint training of the centralized and decentralized policies and prove that AgentMixer converges to an $\epsilon$-approximate Correlated Equilibrium. In the Multi-Agent MuJoCo, SMAC-v2, Matrix Game, and Predator-Prey benchmarks, AgentMixer outperforms or matches state-of-the-art methods.
Code (0)
등록된 구현이 없습니다.
Tasks
Imitation LearningMuJoCoMulti-agent Reinforcement LearningSMACSMAC+Similar Papers 제목 키워드 기반
More Centralized Training, Still Decentralized Execution: Multi-Agent Conditional Policy Factorization
In cooperative multi-agent reinforcement learning (MARL), combining value decomposition with actor-critic enables agents to learn stochastic policies, which are more suitable for the partially observable environment. Giv…
Multi-agent Reinforcement LearningMulti-Agent MDP Homomorphic Networks
This paper introduces Multi-Agent MDP Homomorphic Networks, a class of networks that allows distributed execution using only local information, yet is able to share experience between global symmetries in the joint state…
B3C: A Minimalist Approach to Offline Multi-Agent Reinforcement Learning
Overestimation arising from selecting unseen actions during policy evaluation is a major challenge in offline reinforcement learning (RL). A minimalist approach in the single-agent setting -- adding behavior cloning (BC)…
Multi-agent Reinforcement LearningReinforcement Learning (RL)The Performance Impact of Combining Agent Factorization with Different Learning Algorithms for Multiagent Coordination
Factorizing a multiagent system refers to partitioning the state- action space to individual agents and defining the interactions be- tween those agents. This so-called agent factorization is of much im- portance in r…
ManagementPolicy Gradient MethodsQFree: A Universal Value Function Factorization for Multi-Agent Reinforcement Learning
Centralized training is widely utilized in the field of multi-agent reinforcement learning (MARL) to assure the stability of training process. Once a joint policy is obtained, it is critical to design a value function fa…
Multi-agent Reinforcement Learningreinforcement-learningSMACSMAC++1