Policy Optimization for PDE Control with a Warm Start

2024-03-01 · Xiangyuan Zhang, Saviz Mowlavi, Mouhacine Benosman, Tamer Başar

Dimensionality reduction is crucial for controlling nonlinear partial differential equations (PDE) through a "reduce-then-design" strategy, which identifies a reduced-order model and then implements model-based control solutions. However, inaccuracies in the reduced-order modeling can substantially degrade controller performance, especially in PDEs with chaotic behavior. To address this issue, we augment the reduce-then-design procedure with a policy optimization (PO) step. The PO step fine-tunes the model-based controller to compensate for the modeling error from dimensionality reduction. This augmentation shifts the overall strategy into reduce-then-design-then-adapt, where the model-based controller serves as a warm start for PO. Specifically, we study the state-feedback tracking control of PDEs that aims to align the PDE state with a specific constant target subject to a linear-quadratic cost. Through extensive experiments, we show that a few iterations of PO can significantly improve the model-based controller performance. Our approach offers a cost-effective alternative to PDE control using end-to-end reinforcement learning.

📄 PDF Abstract BibTeX arXiv:2403.01005

Code (0)

등록된 구현이 없습니다.

Tasks

Dimensionality Reduction

Methods 이 논문이 사용한 방법론

PO Stochastic optimization methods have gained significant prominence as effective techniques in contemporary research, addressing complex optimization challenges efficiently. This…

ALIGN In the ALIGN method, visual and language representations are jointly trained from noisy image alt-text data. The image and text encoders are learned via contrastive loss…

Linear Complementarity for Regularized Policy Evaluation and Improvement

2010-12-01 · NeurIPS 2010 12 · Jeffrey Johns, Christopher Painter-Wakefield, Ronald Parr

Recent work in reinforcement learning has emphasized the power of L1 regularization to perform feature selection and prevent overfitting. We propose formulating the L1 regularized linear fixed point problem as a linear c…

feature selection Reinforcement Learning Reinforcement Learning (RL)

Warm Starting of CMA-ES for Contextual Optimization Problems

2025-02-18 · Yuta Sekino, Kento Uchida, Shinichi Shirakawa

Several practical applications of evolutionary computation possess objective functions that receive the design variables and externally given parameters. Such problems are termed contextual optimization problems. These p…

Evolutionary Algorithms

Learning to Warm-Start Fixed-Point Optimization Algorithms

2023-09-14 · Rajiv Sambharya, Georgina Hall, Brandon Amos, Bartolomeo Stellato

We introduce a machine-learning framework to warm-start fixed-point optimization algorithms. Our architecture consists of a neural network mapping problem parameters to warm starts, followed by a predefined number of fix…

Generalization Bounds

Warm-started Semionline Trajectory Planner for Ship's Automatic Docking (Berthing)

2021-12-22 · Dimas M. Rachman, Atsuo Maki, Yoshiki Miyauchi, Naoya Umeda

In the usual framework of control, a reference trajectory is needed as the set point for a feedback controller. This reference trajectory can be generated by solving a trajectory optimization problem. This problem is a c…

Warm-Start Actor-Critic: From Approximation Error to Sub-optimality Gap

2023-06-20 · Hang Wang, Sen Lin, Junshan Zhang

Warm-Start reinforcement learning (RL), aided by a prior policy obtained from offline training, is emerging as a promising RL approach for practical applications. Recent empirical studies have demonstrated that the perfo…

Offline RL Reinforcement Learning (RL)