Decomposed Soft Actor-Critic Method for Cooperative Multi-Agent Reinforcement Learning
Deep reinforcement learning methods have shown great performance on many challenging cooperative multi-agent tasks. Two main promising research directions are multi-agent value function decomposition and multi-agent policy gradients. In this paper, we propose a new decomposed multi-agent soft actor-critic (mSAC) method, which effectively combines the advantages of the aforementioned two methods. The main modules include decomposed Q network architecture, discrete probabilistic policy and counterfactual advantage function (optinal). Theoretically, mSAC supports efficient off-policy learning and addresses credit assignment problem partially in both discrete and continuous action spaces. Tested on StarCraft II micromanagement cooperative multiagent benchmark, we empirically investigate the performance of mSAC against its variants and analyze the effects of the different components. Experimental results demonstrate that mSAC significantly outperforms policy-based approach COMA, and achieves competitive results with SOTA value-based approach Qmix on most tasks in terms of asymptotic perfomance metric. In addition, mSAC achieves pretty good results on large action space tasks, such as 2c_vs_64zg and MMM2.
Code (1)
Tasks
counterfactualDeep Reinforcement LearningMulti-agent Reinforcement Learningreinforcement-learningReinforcement Learning (RL)SMAC+StarcraftStarcraft IISimilar Papers 제목 키워드 기반
Soft Decomposed Policy-Critic: Bridging the Gap for Effective Continuous Control with Discrete RL
Discrete reinforcement learning (RL) algorithms have demonstrated exceptional performance in solving sequential decision tasks with discrete action spaces, such as Atari games. However, their effectiveness is hindered wh…
Atari Gamescontinuous-controlContinuous ControlReinforcement Learning (RL)Developing cooperative policies for multi-stage tasks
This paper proposes the Cooperative Soft Actor Critic (CSAC) method of enabling consecutive reinforcement learning agents to cooperatively solve a long time horizon multi-stage task. This method is achieved by modifying …
Reinforcement Learning (RL)Shared Voxel-Map-Based Cooperative Indoor UAV Guidance with a Multi-Agent Soft Actor-Critic Controller
This paper presents a cooperative indoor UAV guidance framework that combines a shared voxel-map world model with a multi-agent Soft Actor-Critic (MASAC) controller. Multiple drones fuse 360 LiDAR observations into a com…
Continuous ControlFSV: Learning to Factorize Soft Value Function for Cooperative Multi-Agent Reinforcement Learning
We explore energy-based solutions for cooperative multi-agent reinforcement learning (MARL) using the idea of function factorization in centralized training with decentralized execution (CTDE). Existing CTDE based factor…
Multi-agent Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)+2Unveiling Uncertainty-Aware Autonomous Cooperative Learning Based Planning Strategy
In future intelligent transportation systems, autonomous cooperative planning (ACP), becomes a promising technique to increase the effectiveness and security of multi-vehicle interactions. However, multiple uncertainties…
Reinforcement LearningAutonomous VehiclesMotion Planning