Causal Mean Field Multi-Agent Reinforcement Learning

2025-02-20 · Hao Ma, Zhiqiang Pu, Yi Pan, Boyin Liu, Junlong Gao, Zhenyu Guo

Scalability remains a challenge in multi-agent reinforcement learning and is currently under active research. A framework named mean-field reinforcement learning (MFRL) could alleviate the scalability problem by employing the Mean Field Theory to turn a many-agent problem into a two-agent problem. However, this framework lacks the ability to identify essential interactions under nonstationary environments. Causality contains relatively invariant mechanisms behind interactions, though environments are nonstationary. Therefore, we propose an algorithm called causal mean-field Q-learning (CMFQ) to address the scalability problem. CMFQ is ever more robust toward the change of the number of agents though inheriting the compressed representation of MFRL's action-state space. Firstly, we model the causality behind the decision-making process of MFRL into a structural causal model (SCM). Then the essential degree of each interaction is quantified via intervening on the SCM. Furthermore, we design the causality-aware compact representation for behavioral information of agents as the weighted sum of all behavioral information according to their causal effects. We test CMFQ in a mixed cooperative-competitive game and a cooperative game. The result shows that our method has excellent scalability performance in both training in environments containing a large number of agents and testing in environments containing much more agents.

📄 PDF Abstract BibTeX arXiv:2502.14200

Code (0)

등록된 구현이 없습니다.

Tasks

Multi-agent Reinforcement LearningQ-Learningreinforcement-learning Reinforcement Learning

Methods 이 논문이 사용한 방법론

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

Reinforcement Learning in Non-Stationary Discrete-Time Linear-Quadratic Mean-Field Games

2020-09-09 · Muhammad Aneeq uz Zaman, Kaiqing Zhang, Erik Miehling, Tamer Başar

In this paper, we study large population multi-agent reinforcement learning (RL) in the context of discrete-time linear-quadratic mean-field games (LQ-MFGs). Our setting differs from most existing work on RL for MFGs, in…

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning (RL)

Multi Type Mean Field Reinforcement Learning

2020-02-06 · Sriram Ganapathi Subramanian, Pascal Poupart, Matthew E. Taylor, Nidhi Hegde

Mean field theory provides an effective way of scaling multiagent reinforcement learning algorithms to environments with many agents that can be abstracted by a virtual mean agent. In this paper, we extend mean field mul…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Vocal Bursts Type Prediction

Partially Observable Mean Field Multi-Agent Reinforcement Learning Based on Graph-Attention

2023-04-25 · Min Yang, Guanjun Liu, Ziyuan Zhou

Traditional multi-agent reinforcement learning algorithms are difficultly applied in a large-scale multi-agent environment. The introduction of mean field theory has enhanced the scalability of multi-agent reinforcement …

Graph Attention Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning

Causal Multi-Agent Reinforcement Learning: Review and Open Problems

2021-11-12 · St John Grimbly, Jonathan Shock, Arnu Pretorius

This paper serves to introduce the reader to the field of multi-agent reinforcement learning (MARL) and its intersection with methods from the study of causality. We highlight key challenges in MARL and discuss these in …

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Discrete-Time Mean Field Control with Environment States

2021-04-30 · Kai Cui, Anam Tahir, Mark Sinzger, Heinz Koeppl

Multi-agent reinforcement learning methods have shown remarkable potential in solving complex multi-agent problems but mostly lack theoretical guarantees. Recently, mean field control and mean field games have been estab…

Deep Reinforcement Learning Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning+2