Stability-Constrained Markov Decision Processes Using MPC
In this paper, we consider solving discounted Markov Decision Processes (MDPs) under the constraint that the resulting policy is stabilizing. In practice MDPs are solved based on some form of policy approximation. We will leverage recent results proposing to use Model Predictive Control (MPC) as a structured policy in the context of Reinforcement Learning to make it possible to introduce stability requirements directly inside the MPC-based policy. This will restrict the solution of the MDP to stabilizing policies by construction. The stability theory for MPC is most mature for the undiscounted MPC case. Hence, we will first show in this paper that stable discounted MDPs can be reformulated as undiscounted ones. This observation will entail that the MPC-based policy with stability requirements will produce the optimal policy for the discounted MDP if it is stable, and the best stabilizing policy otherwise.
Code (0)
등록된 구현이 없습니다.
Tasks
Model Predictive ControlSimilar Papers 제목 키워드 기반
Functional Stability of Discounted Markov Decision Processes Using Economic MPC Dissipativity Theory
This paper discusses the functional stability of closed-loop Markov Chains under optimal policies resulting from a discounted optimality criterion, forming Markov Decision Processes (MDPs). We investigate the stability o…
Model Predictive ControlQ-LearningvalidEconomic MPC of Markov Decision Processes: Dissipativity in Undiscounted Infinite-Horizon Optimal Control
Economic Model Predictive Control (MPC) dissipativity theory is central to discussing the stability of policies resulting from minimizing economic stage costs. In its current form, the dissipativity theory for economic M…
Model Predictive ControlFinite Horizon Q-learning: Stability, Convergence, Simulations and an application on Smart Grids
Q-learning is a popular reinforcement learning algorithm. This algorithm has however been studied and analysed mainly in the infinite horizon setting. There are several important applications which can be modeled in the …
Q-Learningreinforcement-learningReinforcement Learning (RL)Regular Decision Processes for Grid Worlds
Markov decision processes are typically used for sequential decision making under uncertainty. For many aspects however, ranging from constrained or safe specifications to various kinds of temporal (non-Markovian) depend…
Decision MakingDecision Making Under UncertaintyIncremental LearningSequential Decision MakingConstrained Risk-Averse Markov Decision Processes
We consider the problem of designing policies for Markov decision processes (MDPs) with dynamic coherent risk objectives and constraints. We begin by formulating the problem in a Lagrangian framework. Under the assumptio…