Can Mean Field Control (MFC) Approximate Cooperative Multi Agent Reinforcement Learning (MARL) with Non-Uniform Interaction?

2022-02-28 · Washim Uddin Mondal, Vaneet Aggarwal, Satish V. Ukkusuri

Mean-Field Control (MFC) is a powerful tool to solve Multi-Agent Reinforcement Learning (MARL) problems. Recent studies have shown that MFC can well-approximate MARL when the population size is large and the agents are exchangeable. Unfortunately, the presumption of exchangeability implies that all agents uniformly interact with one another which is not true in many practical scenarios. In this article, we relax the assumption of exchangeability and model the interaction between agents via an arbitrary doubly stochastic matrix. As a result, in our framework, the mean-field `seen' by different agents are different. We prove that, if the reward of each agent is an affine function of the mean-field seen by that agent, then one can approximate such a non-uniform MARL problem via its associated MFC problem within an error of $e=\mathcal{O}(\frac{1}{\sqrt{N}}[\sqrt{|\mathcal{X}|} + \sqrt{|\mathcal{U}|}])$ where $N$ is the population size and $|\mathcal{X}|$, $|\mathcal{U}|$ are the sizes of state and action spaces respectively. Finally, we develop a Natural Policy Gradient (NPG) algorithm that can provide a solution to the non-uniform MARL with an error $\mathcal{O}(\max\{e,\epsilon\})$ and a sample complexity of $\mathcal{O}(\epsilon^{-3})$ for any $\epsilon >0$.

📄 PDF Abstract BibTeX arXiv:2203.00035

Code (1)

https://github.itap.purdue.edu/Clan-labs/NonUniformMFC 공식 구현

Tasks

Multi-agent Reinforcement Learning

Learning Approximate Nash Equilibria in Cooperative Multi-Agent Reinforcement Learning via Mean-Field Subsampling

2026-03-04 · Emile Anand, Ishani Karmarkar arxiv

Many large-scale platforms and networked control systems have a centralized decision maker interacting with a massive population of agents under strict observability constraints. Motivated by such applications, we study …

Multi-agent Reinforcement Learning

Major-Minor Mean Field Multi-Agent Reinforcement Learning

2023-03-19 · Kai Cui, Christian Fabian, Anam Tahir, Heinz Koeppl

Multi-agent reinforcement learning (MARL) remains difficult to scale to many agents. Recent MARL using Mean Field Control (MFC) provides a tractable and rigorous approach to otherwise difficult cooperative MARL. However,…

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning

Mean-Field Controls with Q-learning for Cooperative MARL: Convergence and Complexity Analysis

2020-02-10 · Haotian Gu, Xin Guo, Xiaoli Wei, Renyuan Xu

Multi-agent reinforcement learning (MARL), despite its popularity and empirical success, suffers from the curse of dimensionality. This paper builds the mathematical framework to approximate cooperative MARL by a mean-fi…

Multi-agent Reinforcement Learning Q-Learning Reinforcement Learning

Oracle-free Reinforcement Learning in Mean-Field Games along a Single Sample Path

2022-08-24 · Muhammad Aneeq uz Zaman, Alec Koppel, Sujay Bhatt, Tamer Başar

We consider online reinforcement learning in Mean-Field Games (MFGs). Unlike traditional approaches, we alleviate the need for a mean-field oracle by developing an algorithm that approximates the Mean-Field Equilibrium (…

reinforcement-learning Reinforcement Learning (RL)

Equilibrium pricing of securities in the co-presence of cooperative and non-cooperative populations

2022-09-26 · Masaaki Fujii

In this work, we develop an equilibrium model for price formation of securities in a market composed of two populations of different types: the first one consists of cooperative agents, while the other one consists of no…