Finite-Sample Convergence Bounds for Trust Region Policy Optimization in Mean-Field Games

2025-05-28 · Antonio Ocello, Daniil Tiapkin, Lorenzo Mancini, Mathieu Laurière, Eric Moulines

We introduce Mean-Field Trust Region Policy Optimization (MF-TRPO), a novel algorithm designed to compute approximate Nash equilibria for ergodic Mean-Field Games (MFG) in finite state-action spaces. Building on the well-established performance of TRPO in the reinforcement learning (RL) setting, we extend its methodology to the MFG framework, leveraging its stability and robustness in policy optimization. Under standard assumptions in the MFG literature, we provide a rigorous analysis of MF-TRPO, establishing theoretical guarantees on its convergence. Our results cover both the exact formulation of the algorithm and its sample-based counterpart, where we derive high-probability guarantees and finite sample complexity. This work advances MFG optimization by bridging RL techniques with mean-field decision-making, offering a theoretically grounded approach to solving complex multi-agent problems.

📄 PDF Abstract BibTeX arXiv:2505.22781

Code (0)

등록된 구현이 없습니다.

Tasks

Decision Making Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

TRPO Trust Region Policy Optimization, or TRPO, is a policy gradient method in reinforcement learning that avoids parameter updates that change the policy too much with a KL…

TRSVR: An Adaptive Stochastic Trust-Region Method with Variance Reduction

2026-01-21 · Yuchen Fang, Xinshou Zheng, Javad Lavaei arxiv

We propose a stochastic trust-region method for unconstrained nonconvex optimization that incorporates stochastic variance-reduced gradients (SVRG) to accelerate convergence. Unlike classical trust-region methods, the pr…

A Stochastic Trust Region Method for Non-convex Minimization

2019-03-04 · ICLR 2020 1 · Zebang Shen, Pan Zhou, Cong Fang, Alejandro Ribeiro

We target the problem of finding a local minimum in non-convex finite-sum minimization. Towards this goal, we first prove that the trust region method with inexact gradient and Hessian estimation can achieve a convergenc…

Query-Efficient Quantum Approximate Optimization via Graph-Conditioned Trust Regions

2026-04-27 · Molena Huynh arxiv

In low-depth implementations of the Quantum Approximate Optimization Algorithm (QAOA), the dominant cost is often the number of objective evaluations rather than circuit depth. We introduce a graph-conditioned trust-regi…

Graph Neural Network

A Lyapunov Theory for Finite-Sample Guarantees of Asynchronous Q-Learning and TD-Learning Variants

2021-02-02 · Zaiwei Chen, Siva Theja Maguluri, Sanjay Shakkottai, Karthikeyan Shanmugam

This paper develops an unified framework to study finite-sample convergence guarantees of a large class of value-based asynchronous reinforcement learning (RL) algorithms. We do this by first reformulating the RL algorit…

Q-Learning Reinforcement Learning (RL)

Finite-Sample Analysis of Off-Policy Natural Actor-Critic with Linear Function Approximation

2021-05-26 · Zaiwei Chen, Sajad Khodadadian, Siva Theja Maguluri

In this paper, we develop a novel variant of off-policy natural actor-critic algorithm with linear function approximation and we establish a sample complexity of $\mathcal{O}(\epsilon^{-3})$, outperforming all the previo…