Individual-Level Inverse Reinforcement Learning for Mean Field Games

2022-02-13 · Yang Chen, Libo Zhang, Jiamou Liu, Shuyue Hu

The recent mean field game (MFG) formalism has enabled the application of inverse reinforcement learning (IRL) methods in large-scale multi-agent systems, with the goal of inferring reward signals that can explain demonstrated behaviours of large populations. The existing IRL methods for MFGs are built upon reducing an MFG to a Markov decision process (MDP) defined on the collective behaviours and average rewards of the population. However, this paper reveals that the reduction from MFG to MDP holds only for the fully cooperative setting. This limitation invalidates existing IRL methods on MFGs with non-cooperative environments. To measure more general behaviours in large populations, we study the use of individual behaviours to infer ground-truth reward functions for MFGs. We propose Mean Field IRL (MFIRL), the first dedicated IRL framework for MFGs that can handle both cooperative and non-cooperative environments. Based on this theoretically justified framework, we develop a practical algorithm effective for MFGs with unknown dynamics. We evaluate MFIRL on both cooperative and mixed cooperative-competitive scenarios with many agents. Results demonstrate that MFIRL excels in reward recovery, sample efficiency and robustness in the face of changing dynamics.

📄 PDF Abstract BibTeX arXiv:2202.06401

Code (0)

등록된 구현이 없습니다.

Tasks

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Adversarial Inverse Reinforcement Learning for Mean Field Games

2021-04-29 · Yang Chen, Libo Zhang, Jiamou Liu, Michael Witbrock

Mean field games (MFGs) provide a mathematically tractable framework for modelling large-scale multi-agent systems by leveraging mean field theory to simplify interactions among agents. It enables applying inverse reinfo…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Learning Deep Mean Field Games for Modeling Large Population Behavior

2017-11-08 · ICLR 2018 1 · Jiachen Yang, Xiaojing Ye, Rakshit Trivedi, Huan Xu 외

We consider the problem of representing collective behavior of large populations and predicting the evolution of a population distribution over a discrete state space. A discrete time mean field game (MFG) is motivated a…

Reinforcement Learning

Inverse Concave-Utility Reinforcement Learning is Inverse Game Theory

2024-05-29 · Mustafa Mert Çelikok, Frans A. Oliehoek, Jan-Willem van de Meent

We consider inverse reinforcement learning problems with concave utilities. Concave Utility Reinforcement Learning (CURL) is a generalisation of the standard RL objective, which employs a concave function of the state oc…

Imitation Learning Offline RL reinforcement-learning Reinforcement Learning

Model-Free Mean-Field Reinforcement Learning: Mean-Field MDP and Mean-Field Q-Learning

2019-10-28 · René Carmona, Mathieu Laurière, Zongjun Tan

We study infinite horizon discounted Mean Field Control (MFC) problems with common noise through the lens of Mean Field Markov Decision Processes (MFMDP). We allow the agents to use actions that are randomized not only a…

General Reinforcement Learning Q-Learning Reinforcement Learning Reinforcement Learning (RL)

Maximum Entropy Inverse Reinforcement Learning for Mean-Field Games with Average Reward

2026-06-15 · Şevket Kaan Alkır, Naci Saldı, Berkay Anahtarcı, Can Deha Karıksız arxiv

We study inverse reinforcement learning for discrete-time, infinite-horizon mean-field games (MFGs) under an average-reward criterion. Expert demonstrations are assumed to arise from a stationary mean-field equilibrium u…