On the Gradient Domination of the LQG Problem
We consider solutions to the linear quadratic Gaussian (LQG) regulator problem via policy gradient (PG) methods. Although PG methods have demonstrated strong theoretical guarantees in solving the linear quadratic regulator (LQR) problem, despite its nonconvex landscape, their theoretical understanding in the LQG setting remains limited. Notably, the LQG problem lacks gradient dominance in the classical parameterization, i.e., with a dynamic controller, which hinders global convergence guarantees. In this work, we study PG for the LQG problem by adopting an alternative parameterization of the set of stabilizing controllers and employing a lifting argument. We refer to this parameterization as a history representation of the control input as it is parameterized by past input and output data from the previous p time-steps. This representation enables us to establish gradient dominance and approximate smoothness for the LQG cost. We prove global convergence and per-iteration stability guarantees for policy gradient LQG in model-based and model-free settings. Numerical experiments on an open-loop unstable system are provided to support the global convergence guarantees and to illustrate convergence under different history lengths of the history representation.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Almost sure convergence rates of stochastic gradient methods under gradient domination
Stochastic gradient methods are among the most important algorithms in training machine learning problems. While classical assumptions such as strong convexity allow a simple analysis they are rarely satisfied in applica…
Policy Gradient Methodsreinforcement-learningReinforcement LearningTowards Fast Rates for Federated and Multi-Task Reinforcement Learning
We consider a setting involving $N$ agents, where each agent interacts with an environment modeled as a Markov Decision Process (MDP). The agents' MDPs differ in their reward functions, capturing heterogeneous objectives…
reinforcement-learningReinforcement LearningFull error analysis of policy gradient learning algorithms for exploratory linear quadratic mean-field control problem in continuous time with common noise
We consider reinforcement learning (RL) methods for finding optimal policies in linear quadratic (LQ) mean field control (MFC) problems over an infinite horizon in continuous time, with common noise and entropy regulariz…
Reinforcement Learning (RL)Matrix Domination: Convergence of a Genetic Algorithm Metaheuristic with the Wisdom of Crowds to Solve the NP-Complete Problem
This research explores the application of a genetic algorithm metaheuristic enriched by the wisdom of crowds in order to address the NP-Complete matrix domination problem (henceforth: TMDP) which is itself a constraint o…
Decision MakingBetter Runtime Guarantees Via Stochastic Domination
Apart from few exceptions, the mathematical runtime analysis of evolutionary algorithms is mostly concerned with expected runtimes. In this work, we argue that stochastic domination is a notion that should be used more f…
Evolutionary Algorithms