Greedy-based Value Representation for Efficient Coordination in Multi-agent Reinforcement Learning

2021-09-29 · Lipeng Wan, Zeyang Liu, Xingyu Chen, Han Wang, Xuguang Lan

Due to the representation limitation of the joint Q value function, multi-agent reinforcement learning (MARL) methods with linear or monotonic value decomposition can not ensure the optimal consistency (i.e. the correspondence between the individual greedy actions and the maximal true Q value), leading to instability and poor coordination. Existing methods focus on addressing the representation limitation through learning the complete expressiveness, which is impractical and may deteriorate the performance in complex tasks. In this paper, we introduce the True-Global-Max (TGM) condition for linear and monotonic value decomposition to achieve the optimal consistency directly, where the TGM condition can be ensured under the unique stability of the optimal greedy action. Therefore, we propose the greedy-based value representation (GVR), which stabilises the optimal greedy action via inferior target shaping and destabilises non-optimal greedy actions via superior experience replay. We conduct experiments on various benchmarks, where GVR significantly outperforms state-of-the-art baselines. Experiment results demonstrate that our method can meet the optimal consistency under sufficient exploration and is more efficient than methods with complete expressiveness capability.

📄 PDF Abstract BibTeX

Code (0)

등록된 구현이 없습니다.

Tasks

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning (RL)

Non-Linear Coordination Graphs

2022-10-26 · Yipeng Kang, Tonghan Wang, Xiaoran Wu, Qianlan Yang 외

Value decomposition multi-agent reinforcement learning methods learn the global value function as a mixing of each agent's individual utility functions. Coordination graphs (CGs) represent a higher-order decomposition by…

Multi-agent Reinforcement Learning

Greedy based Value Representation for Optimal Coordination in Multi-agent Reinforcement Learning

2022-11-22 · Lipeng Wan, Zeyang Liu, Xingyu Chen, Xuguang Lan 외

Due to the representation limitation of the joint Q value function, multi-agent reinforcement learning methods with linear value decomposition (LVD) or monotonic value decomposition (MVD) suffer from relative overgeneral…

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning (RL)

Self-Organized Polynomial-Time Coordination Graphs

2021-12-07 · Qianlan Yang, Weijun Dong, Zhizhou Ren, Jianhao Wang 외

Coordination graph is a promising approach to model agent collaboration in multi-agent reinforcement learning. It conducts a graph-based value factorization and induces explicit coordination among agents to complete comp…

Computational Efficiency Multi-agent Reinforcement Learning

Ensemble Value Functions for Efficient Exploration in Multi-Agent Reinforcement Learning

2023-02-07 · Lukas Schäfer, Oliver Slumbers, Stephen Mcaleer, Yali Du 외

Multi-agent reinforcement learning (MARL) requires agents to explore within a vast joint action space to find joint actions that lead to coordination. Existing value-based MARL algorithms commonly rely on random explorat…

Efficient Exploration Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning (RL)

Adaptive Value Decomposition with Greedy Marginal Contribution Computation for Cooperative Multi-Agent Reinforcement Learning

2023-02-14 · Shanqi Liu, Yujing Hu, Runze Wu, Dong Xing 외

Real-world cooperation often requires intensive coordination among agents simultaneously. This task has been extensively studied within the framework of cooperative multi-agent reinforcement learning (MARL), and value de…