QR-MIX: Distributional Value Function Factorisation for Cooperative Multi-Agent Reinforcement Learning

2020-09-09 · Jian Hu, Seth Austin Harding, Haibin Wu, Siyue Hu, Shih-wei Liao

In Cooperative Multi-Agent Reinforcement Learning (MARL) and under the setting of Centralized Training with Decentralized Execution (CTDE), agents observe and interact with their environment locally and independently. With local observation and random sampling, the randomness in rewards and observations leads to randomness in long-term returns. Existing methods such as Value Decomposition Network (VDN) and QMIX estimate the value of long-term returns as a scalar that does not contain the information of randomness. Our proposed model QR-MIX introduces quantile regression, modeling joint state-action values as a distribution, combining QMIX with Implicit Quantile Network (IQN). However, the monotonicity in QMIX limits the expression of joint state-action value distribution and may lead to incorrect estimation results in non-monotonic cases. Therefore, we proposed a flexible loss function to approximate the monotonicity found in QMIX. Our model is not only more tolerant of the randomness of returns, but also more tolerant of the randomness of monotonic constraints. The experimental results demonstrate that QR-MIX outperforms the previous state-of-the-art method QMIX in the StarCraft Multi-Agent Challenge (SMAC) environment.

📄 PDF Abstract BibTeX arXiv:2009.04197

Code (0)

등록된 구현이 없습니다.

Tasks

Multi-agent Reinforcement Learningquantile regressionreinforcement-learning Reinforcement Learning (RL)SMAC SMAC+Starcraft

QGNN: Value Function Factorisation with Graph Neural Networks

2022-05-25 · Ryan Kortvelesy, Amanda Prorok

In multi-agent reinforcement learning, the use of a global objective is a powerful tool for incentivising cooperation. Unfortunately, it is not sample-efficient to train individual agents with a global reward, because it…

Graph Neural Network Multi-agent Reinforcement Learning Starcraft

MMD-MIX: Value Function Factorisation with Maximum Mean Discrepancy for Cooperative Multi-Agent Reinforcement Learning

2021-06-22 · Zhiwei Xu, Dapeng Li, Yunpeng Bai, Guoliang Fan

In the real world, many tasks require multiple agents to cooperate with each other under the condition of local observations. To solve such problems, many multi-agent reinforcement learning methods based on Centralized T…

Distributional Reinforcement Learning Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning+4

The challenge of redundancy on multi-agent value factorisation

2023-03-28 · Siddarth Singh, Benjamin Rosman

In the field of cooperative multi-agent reinforcement learning (MARL), the standard paradigm is the use of centralised training and decentralised execution where a central critic conditions the policies of the cooperativ…

Multi-agent Reinforcement Learning

DFAC Framework: Factorizing the Value Function via Quantile Mixture for Multi-Agent Distributional Q-Learning

2021-02-16 · Wei-Fang Sun, Cheng-Kuang Lee, Chun-Yi Lee

In fully cooperative multi-agent reinforcement learning (MARL) settings, the environments are highly stochastic due to the partial observability of each agent and the continuously changing policies of the other agents. T…

Multi-agent Reinforcement Learning Q-Learning SMAC SMAC++1

A Unified Framework for Factorizing Distributional Value Functions for Multi-Agent Reinforcement Learning

2023-06-04 · Wei-Fang Sun, Cheng-Kuang Lee, Simon See, Chun-Yi Lee

In fully cooperative multi-agent reinforcement learning (MARL) settings, environments are highly stochastic due to the partial observability of each agent and the continuously changing policies of other agents. To addres…

Multi-agent Reinforcement Learning reinforcement-learning SMAC Starcraft