Greedy based Value Representation for Optimal Coordination in Multi-agent Reinforcement Learning

2022-11-22 · Lipeng Wan, Zeyang Liu, Xingyu Chen, Xuguang Lan, Nanning Zheng

Due to the representation limitation of the joint Q value function, multi-agent reinforcement learning methods with linear value decomposition (LVD) or monotonic value decomposition (MVD) suffer from relative overgeneralization. As a result, they can not ensure optimal consistency (i.e., the correspondence between individual greedy actions and the maximal true Q value). In this paper, we derive the expression of the joint Q value function of LVD and MVD. According to the expression, we draw a transition diagram, where each self-transition node (STN) is a possible convergence. To ensure optimal consistency, the optimal node is required to be the unique STN. Therefore, we propose the greedy-based value representation (GVR), which turns the optimal node into an STN via inferior target shaping and further eliminates the non-optimal STNs via superior experience replay. In addition, GVR achieves an adaptive trade-off between optimality and stability. Our method outperforms state-of-the-art baselines in experiments on various benchmarks. Theoretical proofs and empirical results on matrix games demonstrate that GVR ensures optimal consistency under sufficient exploration.

📄 PDF Abstract BibTeX arXiv:2211.12075

Code (0)

등록된 구현이 없습니다.

Tasks

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning (RL)

Greedy-based Value Representation for Efficient Coordination in Multi-agent Reinforcement Learning

2021-09-29 · Lipeng Wan, Zeyang Liu, Xingyu Chen, Han Wang 외

Due to the representation limitation of the joint Q value function, multi-agent reinforcement learning (MARL) methods with linear or monotonic value decomposition can not ensure the optimal consistency (i.e. the correspo…

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning (RL)

Non-Linear Coordination Graphs

2022-10-26 · Yipeng Kang, Tonghan Wang, Xiaoran Wu, Qianlan Yang 외

Value decomposition multi-agent reinforcement learning methods learn the global value function as a mixing of each agent's individual utility functions. Coordination graphs (CGs) represent a higher-order decomposition by…

Multi-agent Reinforcement Learning

Resolving Implicit Coordination in Multi-Agent Deep Reinforcement Learning with Deep Q-Networks & Game Theory

2020-12-08 · Griffin Adams, Sarguna Janani Padmanabhan, Shivang Shekhar

We address two major challenges of implicit coordination in multi-agent deep reinforcement learning: non-stationarity and exponential growth of state-action space, by combining Deep-Q Networks for policy learning with Na…

Deep Reinforcement Learning OpenAI Gym Reinforcement Learning (RL)

Parallel Double Greedy Submodular Maximization

2014-12-01 · NeurIPS 2014 12 · Xinghao Pan, Stefanie Jegelka, Joseph E. Gonzalez, Joseph K. Bradley 외

Many machine learning problems can be reduced to the maximization of submodular functions. Although well understood in the serial setting, the parallel maximization of submodular functions remains an open area of researc…

Adaptive Value Decomposition with Greedy Marginal Contribution Computation for Cooperative Multi-Agent Reinforcement Learning

2023-02-14 · Shanqi Liu, Yujing Hu, Runze Wu, Dong Xing 외

Real-world cooperation often requires intensive coordination among agents simultaneously. This task has been extensively studied within the framework of cooperative multi-agent reinforcement learning (MARL), and value de…