DeCOM: Decomposed Policy for Constrained Cooperative Multi-Agent Reinforcement Learning
In recent years, multi-agent reinforcement learning (MARL) has presented impressive performance in various applications. However, physical limitations, budget restrictions, and many other factors usually impose \textit{constraints} on a multi-agent system (MAS), which cannot be handled by traditional MARL frameworks. Specifically, this paper focuses on constrained MASes where agents work \textit{cooperatively} to maximize the expected team-average return under various constraints on expected team-average costs, and develops a \textit{constrained cooperative MARL} framework, named DeCOM, for such MASes. In particular, DeCOM decomposes the policy of each agent into two modules, which empowers information sharing among agents to achieve better cooperation. In addition, with such modularization, the training algorithm of DeCOM separates the original constrained optimization into an unconstrained optimization on reward and a constraints satisfaction problem on costs. DeCOM then iteratively solves these problems in a computationally efficient manner, which makes DeCOM highly scalable. We also provide theoretical guarantees on the convergence of DeCOM's policy update algorithm. Finally, we validate the effectiveness of DeCOM with various types of costs in both toy and large-scale (with 500 agents) environments.
Code (4)
Tasks
Multi-agent Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Decomposed Soft Actor-Critic Method for Cooperative Multi-Agent Reinforcement Learning
Deep reinforcement learning methods have shown great performance on many challenging cooperative multi-agent tasks. Two main promising research directions are multi-agent value function decomposition and multi-agent poli…
counterfactualDeep Reinforcement LearningMulti-agent Reinforcement Learningreinforcement-learning+4Distributed Multi Robot Lunar Cargo Transportation via Phase Decomposed Reinforcement Learning
Modular reconfigurable robotic systems provide a scalable solution for cooperative surface operations in future lunar missions. However, cooperative cargo transportation remains challenging due to morphology-dependent to…
Reinforcement LearningDecision MakingPolicy Diagnosis via Measuring Role Diversity in Cooperative Multi-agent RL
Cooperative multi-agent reinforcement learning (MARL) is making rapid progress for solving tasks in a grid world and real-world scenarios, in which agents are given different attributes and goals, resulting in different …
DiversityMulti-agent Reinforcement LearningSMACSMAC++1Role Diversity Matters: A Study of Cooperative Training Strategies for Multi-Agent RL
Cooperative multi-agent reinforcement learning (MARL) is making rapid progress for solving tasks in a grid world and real-world scenarios, in which agents are given different attributes and goals. For example, in Starcra…
DiversityMulti-agent Reinforcement LearningSMACSMAC++2CORD: Generalizable Cooperation via Role Diversity
Cooperative multi-agent reinforcement learning (MARL) aims to develop agents that can collaborate effectively. However, most cooperative MARL methods overfit training agents, making learned policies not generalize well t…
DiversityMulti-agent Reinforcement Learning