Fast Policy Learning for Linear Quadratic Control with Entropy Regularization

2023-11-23 · Xin Guo, Xinyu Li, Renyuan Xu

This paper proposes and analyzes two new policy learning methods: regularized policy gradient (RPG) and iterative policy optimization (IPO), for a class of discounted linear-quadratic control (LQC) problems over an infinite time horizon with entropy regularization. Assuming access to the exact policy evaluation, both proposed approaches are proven to converge linearly in finding optimal policies of the regularized LQC. Moreover, the IPO method can achieve a super-linear convergence rate once it enters a local region around the optimal policy. Finally, when the optimal policy for an RL problem with a known environment is appropriately transferred as the initial policy to an RL problem with an unknown environment, the IPO method is shown to enable a super-linear convergence rate if the two environments are sufficiently close. Performances of these proposed algorithms are supported by numerical examples.

📄 PDF Abstract BibTeX arXiv:2311.14168

Code (0)

등록된 구현이 없습니다.

Optimal scheduling of entropy regulariser for continuous-time linear-quadratic reinforcement learning

2022-08-08 · Lukasz Szpruch, Tanut Treetanthiploet, Yufei Zhang

This work uses the entropy-regularised relaxed stochastic control perspective as a principled framework for designing reinforcement learning (RL) algorithms. Herein agent interacts with the environment by generating nois…

reinforcement-learning Reinforcement Learning (RL)Scheduling

Global Convergence of Policy Gradient for Entropy Regularized Linear-Quadratic Control with Multiplicative Noise

2025-10-03 · Gabriel Diaz, Lucky Li, Wenhao Zhang arxiv

Reinforcement Learning (RL) has emerged as a powerful framework for sequential decision-making in dynamic environments, particularly when system parameters are unknown. This paper investigates RL-based control for entrop…

Reinforcement Learning

Reinforcement Learning for a Discrete-Time Linear-Quadratic Control Problem with an Application

2024-12-08 · Lucky Li

We study the discrete-time linear-quadratic (LQ) control model using reinforcement learning (RL). Using entropy to measure the cost of exploration, we prove that the optimal feedback policy for the problem must be Gaussi…

Management Reinforcement Learning (RL)

Regret Analysis of Certainty Equivalence Policies in Continuous-Time Linear-Quadratic Systems

2022-06-09 · Mohamad Kazem Shirani Faradonbeh

This work theoretically studies a ubiquitous reinforcement learning policy for controlling the canonical model of continuous-time stochastic linear-quadratic systems. We show that randomized certainty equivalent policy a…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Convergence of policy gradient methods for finite-horizon exploratory linear-quadratic control problems

2022-11-01 · Michael Giegrich, Christoph Reisinger, Yufei Zhang

We study the global linear convergence of policy gradient (PG) methods for finite-horizon continuous-time exploratory linear-quadratic control (LQC) problems. The setting includes stochastic LQC problems with indefinite …

Policy Gradient Methods