Model-free optimal controller for discrete-time Markovian jump linear systems: A Q-learning approach
This research paper introduces a model-free optimal controller for discrete-time Markovian jump linear systems (MJLSs), employing principles from the methodology of reinforcement learning (RL). While Q-learning methods have demonstrated efficacy in determining optimal controller gains for deterministic systems, their application to systems with Markovian switching remains unexplored. To address this research gap, we propose a Q-function involving the Markovian mode. Subsequently, a Q-learning algorithm is proposed to learn the unknown kernel matrix using raw input-state information from the system. Notably, the study proves the convergence of the proposed Q-learning optimal controller gains to the model-based optimal controller gains after proving the convergence of a value iteration algorithm as the first step. Addition of excitation noise to input which is required to ensure the leaning performance does not lead to any bias. Unlike the conventional optimal controller, the proposed method does not require any knowledge on system dynamics and eliminates the need for solving coupled algebraic Riccati equations arising in optimal control of MJLSs. Finally, the efficiency of the proposed method is demonstrated through a simulation study.
Code (0)
등록된 구현이 없습니다.
Tasks
Q-LearningReinforcement Learning (RL)Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Policy Learning of MDPs with Mixed Continuous/Discrete Variables: A Case Study on Model-Free Control of Markovian Jump Systems
Markovian jump linear systems (MJLS) are an important class of dynamical systems that arise in many control applications. In this paper, we introduce the problem of controlling unknown (discrete-time) MJLS as a new bench…
Policy Optimization for Markovian Jump Linear Quadratic Control: Gradient-Based Methods and Global Convergence
Recently, policy optimization for control purposes has received renewed attention due to the increasing interest in reinforcement learning. In this paper, we investigate the global convergence of gradient-based policy op…
Policy Gradient MethodsCompositional Controller Synthesis for Interconnected Stochastic Systems with Markovian Switching
In this work, we propose a compositional scheme for the safety controller synthesis of interconnected discrete-time stochastic systems with Markovian switching signals. Our proposed approach is based on a notion of so-ca…
Predictive Control of Linear Discrete-Time Markovian Jump Systems by Learning Recurrent Patterns
Incorporating pattern-learning for prediction (PLP) in many discrete-time or discrete-event systems allows for computation-efficient controller design by memorizing patterns to schedule control policies based on their fu…
Deep Recurrent Optimal Stopping
Deep neural networks (DNNs) have recently emerged as a powerful paradigm for solving Markovian optimal stopping problems. However, a ready extension of DNN-based methods to non-Markovian settings requires significant sta…