Reinforcement Learning for Finite Space Mean-Field Type Games
Mean field type games (MFTGs) describe Nash equilibria between large coalitions: each coalition consists of a continuum of cooperative agents who maximize the average reward of their coalition while interacting non-cooperatively with a finite number of other coalitions. Although the theory has been extensively developed, we are still lacking efficient and scalable computational methods. Here, we develop reinforcement learning methods for such games in a finite space setting with general dynamics and reward functions. We start by proving that MFTG solution yields approximate Nash equilibria in finite-size coalition games. We then propose two algorithms. The first is based on quantization of mean-field spaces and Nash Q-learning. We provide convergence and stability analysis. We then propose a deep reinforcement learning algorithm, which can scale to larger spaces. Numerical experiments in 5 environments with mean-field distributions of dimension up to $200$ show the scalability and efficiency of the proposed method.
Code (0)
등록된 구현이 없습니다.
Tasks
Deep Reinforcement LearningQ-LearningQuantizationreinforcement-learningReinforcement LearningSimilar Papers 제목 키워드 기반
Model Free Reinforcement Learning Algorithm for Stationary Mean field Equilibrium for Multiple Types of Agents
We consider a multi-agent Markov strategic interaction over an infinite horizon where agents can be of multiple types. We model the strategic interaction as a mean-field game in the asymptotic limit when the number of ag…
Reinforcement Learning (RL)Deep Reinforcement Learning for Infinite Horizon Mean Field Problems in Continuous Spaces
We present the development and analysis of a reinforcement learning (RL) algorithm designed to solve continuous-space mean field game (MFG) and mean field control (MFC) problems in a unified manner. The proposed approach…
Deep Reinforcement LearningReinforcement Learning (RL)Dynamic mean field programming
A dynamic mean field theory is developed for finite state and action Bayesian reinforcement learning in the large state space limit. In an analogy with statistical physics, the Bellman equation is studied as a disordered…
reinforcement-learningReinforcement LearningConvergence of Actor-Critic Learning for Mean Field Games and Mean Field Control in Continuous Spaces
We establish the convergence of the deep actor-critic reinforcement learning algorithm presented in [Angiuli et al., 2023a] in the setting of continuous state and action spaces with an infinite discrete-time horizon. Thi…
Reinforcement LearningFinite-Sample Convergence Bounds for Trust Region Policy Optimization in Mean-Field Games
We introduce Mean-Field Trust Region Policy Optimization (MF-TRPO), a novel algorithm designed to compute approximate Nash equilibria for ergodic Mean-Field Games (MFG) in finite state-action spaces. Building on the well…
Decision MakingReinforcement Learning (RL)