Policy Gradient Methods for Designing Dynamic Output Feedback Controllers
This paper proposes model-based and model-free policy gradient methods (PGMs) for designing dynamic output feedback controllers for discrete-time partially observable systems. To fulfill this objective, we first show that any dynamic output feedback controller design is equivalent to a state-feedback controller design for a newly introduced system whose internal state is a finite-length input-output history (IOH). Next, based on this equivalency, we propose a model-based PGM and show its global linear convergence by proving that the Polyak-Lojasiewicz inequality holds for a reachability-based lossless projection of the IOH dynamics. Moreover, we propose two model-free implementations of the PGM: the multi- and single-episodic PGM. The former is a Monte Carlo approximation of the model-based PGM, whereas the latter is a simplified version of the former for ease of use in real systems. A sample complexity analysis of both methods is also presented. Finally, the effectiveness of the model-based/model-free PGMs is investigated through a numerical simulation.
Code (0)
등록된 구현이 없습니다.
Tasks
Policy Gradient MethodsMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Sparse optimal control of networks with multiplicative noise via policy gradient
We give algorithms for designing near-optimal sparse controllers using policy gradient with applications to control of systems corrupted by multiplicative noise, which is increasingly important in emerging complex dynami…
Deterministic Policy Gradient Primal-Dual Methods for Continuous-Space Constrained MDPs
We study the problem of computing deterministic optimal policies for constrained Markov decision processes (MDPs) with continuous state and action spaces, which are widely encountered in constrained dynamical systems. De…
continuous-controlContinuous ControlPolicy Gradient MethodsRobot NavigationOn the Optimization Landscape of Dynamic Output Feedback: A Case Study for Linear Quadratic Regulator
The convergence of policy gradient algorithms in reinforcement learning hinges on the optimization landscape of the underlying optimal control problem. Theoretical insights into these algorithms can often be acquired fro…
Decision MakingPolicy Gradient MethodsModel-Free Output Feedback Stabilization via Policy Gradient Methods
Stabilizing a dynamical system is a fundamental problem that serves as a cornerstone for many complex tasks in the field of control systems. The problem becomes challenging when the system model is unknown. Among the Rei…
Reinforcement LearningWhy Semantic Entropy Fails: Geometry-Aware and Calibrated Uncertainty for Policy Optimization
Post-training has become central to improving reasoning and alignment in large language models, where critic-free models enable scalable learning from model-generated outputs but lack principled mechanisms to distinguish…