Value Functions Factorization with Latent State Information Sharing in Decentralized Multi-Agent Policy Gradients
Value function factorization via centralized training and decentralized execution is promising for solving cooperative multi-agent reinforcement tasks. One of the approaches in this area, QMIX, has become state-of-the-art and achieved the best performance on the StarCraft II micromanagement benchmark. However, the monotonic-mixing of per agent estimates in QMIX is known to restrict the joint action Q-values it can represent, as well as the insufficient global state information for single agent value function estimation, often resulting in suboptimality. To this end, we present LSF-SAC, a novel framework that features a variational inference-based information-sharing mechanism as extra state information to assist individual agents in the value function factorization. We demonstrate that such latent individual state information sharing can significantly expand the power of value function factorization, while fully decentralized execution can still be maintained in LSF-SAC through a soft-actor-critic design. We evaluate LSF-SAC on the StarCraft II micromanagement challenge and demonstrate that it outperforms several state-of-the-art methods in challenging collaborative tasks. We further set extensive ablation studies for locating the key factors accounting for its performance improvements. We believe that this new insight can lead to new local value estimation methods and variational deep learning algorithms. A demo video and code of implementation can be found at https://sites.google.com/view/sacmm.
Code (1)
Tasks
StarcraftStarcraft IIVariational InferenceSimilar Papers 제목 키워드 기반
On Stateful Value Factorization in Multi-Agent Reinforcement Learning
Value factorization is a popular paradigm for designing scalable multi-agent reinforcement learning algorithms. However, current factorization methods make choices without full justification that may limit their performa…
Multi-agent Reinforcement Learningreinforcement-learningReinforcement LearningStarcraft+1Gaussian Process Latent Variable Model Factorization for Context-aware Recommender Systems
Context-aware recommender systems (CARS) have gained increasing attention due to their ability to utilize contextual information. Compared to traditional recommender systems, CARS are, in general, able to generate more a…
Dimensionality ReductionRecommendation SystemsSCFM: Social and crowdsourcing factorization machines for recommendation
With the rapid development of social networks, the exponential growth of social information has attracted much attention. Social information has great value in recommender systems to alleviate the sparsity and cold sta…
Recommendation SystemsReMIX: Regret Minimization for Monotonic Value Function Factorization in Multiagent Reinforcement Learning
Value function factorization methods have become a dominant approach for cooperative multiagent reinforcement learning under a centralized training and decentralized execution paradigm. By factorizing the optimal joint a…
Decision Makingreinforcement-learningReinforcement Learning (RL)StarcraftLearning Nearly Decomposable Value Functions Via Communication Minimization
Reinforcement learning encounters major challenges in multi-agent settings, such as scalability and non-stationarity. Recently, value function factorization learning emerges as a promising way to address these challenges…
Reinforcement LearningStarcraft