Deterministic policy gradient based optimal control with probabilistic constraints
This paper studies a deep deterministic policy gradient (DDPG) based actor critic (AC) reinforcement learning (RL) technique to control a linear discrete-time system with a quadratic control cost while ensuring a constraint on the probability of potentially risky or undesirable events. The proposed methodology can be applied to both known and unknown system models with minor adjustments to the reward structure (negative cost). The problem is formulated by considering the average expected quadratic cost of the states and inputs over an infinite time horizon. Risky or undesirable events are represented as functions of the states at the next time step exceeding a user-defined limit. Two strategies are employed to manage the probabilistic constraint in scenarios of known and unknown system models. In the case of a known system model, the probabilistic constraint is replaced with an upper bound, such as the Chernoff bound. For unknown system models, the expected value of the indicator function of the occurrence of the risky or undesirable event is used. We have adopted a deterministic policy gradient (DPG) based AC method to derive a parameterised optimal policy. Extensive numerical simulations are performed using a second- and a fourth-order system, and the proposed method is compared with the standard risk-neutral linear quadratic regulator (LQR) and a chance-constrained model predictive control (MPC) method. The results demonstrate the effectiveness of the proposed approach in both known and unknown system model scenarios.
Code (0)
등록된 구현이 없습니다.
Tasks
Model Predictive Controlreinforcement-learningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Policy Gradient-based Model Free Optimal LQG Control with a Probabilistic Risk Constraint
In this paper, we investigate a model-free optimal control design that minimizes an infinite horizon average expected quadratic cost of states and control actions subject to a probabilistic risk or chance constraint usin…
Model Predictive ControlDeterministic Trajectory Optimization through Probabilistic Optimal Control
In this article, we discuss two algorithms tailored to discrete-time deterministic finite-horizon nonlinear optimal control problems or so-called deterministic trajectory optimization problems. Both algorithms can be der…
Uncertainty QuantificationDeterministic Policy Gradient Primal-Dual Methods for Continuous-Space Constrained MDPs
We study the problem of computing deterministic optimal policies for constrained Markov decision processes (MDPs) with continuous state and action spaces, which are widely encountered in constrained dynamical systems. De…
continuous-controlContinuous ControlPolicy Gradient MethodsRobot NavigationRobust Deterministic Policy Gradient for Disturbance Attenuation and Its Application to Quadrotor Control
Practical control systems pose significant challenges in identifying optimal control policies due to uncertainties in the system model and external disturbances. While $H_\infty$ control techniques are commonly used to d…
Deep Reinforcement LearningActor-Critic Reinforcement Learning with Phased Actor
Policy gradient methods in actor-critic reinforcement learning (RL) have become perhaps the most promising approaches to solving continuous optimal control problems. However, the trial-and-error nature of RL and the inhe…
Policy Gradient Methodsreinforcement-learningReinforcement LearningReinforcement Learning (RL)