The Feasibility of Constrained Reinforcement Learning Algorithms: A Tutorial Study
Satisfying safety constraints is a priority concern when solving optimal control problems (OCPs). Due to the existence of infeasibility phenomenon, where a constraint-satisfying solution cannot be found, it is necessary to identify a feasible region before implementing a policy. Existing feasibility theories built for model predictive control (MPC) only consider the feasibility of optimal policy. However, reinforcement learning (RL), as another important control method, solves the optimal policy in an iterative manner, which comes with a series of non-optimal intermediate policies. Feasibility analysis of these non-optimal policies is also necessary for iteratively improving constraint satisfaction; but that is not available under existing MPC feasibility theories. This paper proposes a feasibility theory that applies to both MPC and RL by filling in the missing part of feasibility analysis for an arbitrary policy. The basis of our theory is to decouple policy solving and implementation into two temporal domains: virtual-time domain and real-time domain. This allows us to separately define initial and endless, state and policy feasibility, and their corresponding feasible regions. Based on these definitions, we analyze the containment relationships between different feasible regions, which enables us to describe the feasible region of an arbitrary policy. We further provide virtual-time constraint design rules along with a practical design tool called feasibility function that helps to achieve the maximum feasible region. We review most of existing constraint formulations and point out that they are essentially applications of feasibility functions in different forms. We demonstrate our feasibility theory by visualizing different feasible regions under both MPC and RL policies in an emergency braking control task.
Code (0)
등록된 구현이 없습니다.
Tasks
Model Predictive Controlreinforcement-learningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Implementation of Linear Model Predictive Control -- Tutorial
This tutorial shows an overview of Model Predictive Control with a linear discrete-time system and constrained states and inputs. The focus is on the implementation of the method under consideration of stability and recu…
modelModel Predictive ControlBenchmarking Actor-Critic Deep Reinforcement Learning Algorithms for Robotics Control with Action Constraints
This study presents a benchmark for evaluating action-constrained reinforcement learning (RL) algorithms. In action-constrained RL, each action taken by the learning system must comply with certain constraints. These con…
BenchmarkingDeep Reinforcement LearningReinforcement Learning (RL)KKT Conditions, First-Order and Second-Order Optimization, and Distributed Optimization: Tutorial and Survey
This is a tutorial and survey paper on Karush-Kuhn-Tucker (KKT) conditions, first-order and second-order numerical optimization, and distributed optimization. After a brief review of history of optimization, we start wit…
Distributed OptimizationMultiobjective OptimizationSecond-order methodsStochastic OptimizationOffline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
In this tutorial article, we aim to provide the reader with the conceptual tools needed to get started on research on offline reinforcement learning algorithms: reinforcement learning algorithms that utilize previously c…
Decision MakingDeep Reinforcement Learningreinforcement-learningReinforcement Learning+1A Tutorial Introduction to Reinforcement Learning
In this paper, we present a brief survey of Reinforcement Learning (RL), with particular emphasis on Stochastic Approximation (SA) as a unifying theme. The scope of the paper includes Markov Reward Processes, Markov Deci…
Q-Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)+1