Policy Optimization for Continuous-time Linear-Quadratic Graphon Mean Field Games
Multi-agent reinforcement learning, despite its popularity and empirical success, faces significant scalability challenges in large-population dynamic games. Graphon mean field games (GMFGs) offer a principled framework for approximating such games while capturing heterogeneity among players. In this paper, we propose and analyze a policy optimization framework for continuous-time, finite-horizon linear-quadratic GMFGs. Exploiting the structural properties of GMFGs, we design an efficient policy parameterization in which each player's policy is represented as an affine function of their private state, with a shared slope function and player-specific intercepts. We develop a bilevel optimization algorithm that alternates between policy gradient updates for best-response computation under a fixed population distribution, and distribution updates using the resulting policies. We prove linear convergence of the policy gradient steps to best-response policies and establish global convergence of the overall algorithm to the Nash equilibrium. The analysis relies on novel landscape characterizations over infinite-dimensional policy spaces. Numerical experiments demonstrate the convergence and robustness of the proposed algorithm under varying graphon structures, noise levels, and action frequencies.
Code (0)
등록된 구현이 없습니다.
Tasks
Bilevel OptimizationMulti-agent Reinforcement LearningSimilar Papers 제목 키워드 기반
Continuous-time iterative linear-quadratic regulator
We present a continuous-time equivalent to the well-known iterative linear-quadratic algorithm including an implementation of a backtracking line-search policy and a novel regularization approach based on the necessary c…
Numerical IntegrationRegret Analysis of Certainty Equivalence Policies in Continuous-Time Linear-Quadratic Systems
This work theoretically studies a ubiquitous reinforcement learning policy for controlling the canonical model of continuous-time stochastic linear-quadratic systems. We show that randomized certainty equivalent policy a…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Convergence of policy gradient methods for finite-horizon exploratory linear-quadratic control problems
We study the global linear convergence of policy gradient (PG) methods for finite-horizon continuous-time exploratory linear-quadratic control (LQC) problems. The setting includes stochastic LQC problems with indefinite …
Policy Gradient MethodsGlobal Convergence of Policy Gradient for Linear-Quadratic Mean-Field Control/Game in Continuous Time
Reinforcement learning is a powerful tool to learn the optimal policy of possibly multiple agents by interacting with the environment. As the number of agents grow to be very large, the system can be approximated by a me…
Derivative-Free Policy Optimization for Linear Risk-Sensitive and Robust Control Design: Implicit Regularization and Sample Complexity
Direct policy search serves as one of the workhorses in modern reinforcement learning (RL), and its applications in continuous control tasks have recently attracted increasing attention. In this work, we investigate the …
continuous-controlContinuous ControlMulti-agent Reinforcement Learningreinforcement-learning+2