Dynamic Programming-based Approximate Optimal Control for Model-Based Reinforcement Learning

2023-12-22 · Prakash Mallick, Zhiyong Chen

This article proposes an improved trajectory optimization approach for stochastic optimal control of dynamical systems affected by measurement noise by combining optimal control with maximum likelihood techniques to improve the reduction of the cumulative cost-to-go. A modified optimization objective function that incorporates dynamic programming-based controller design is presented to handle the noise in the system and sensors. Empirical results demonstrate the effectiveness of the approach in reducing stochasticity and allowing for an intermediate step to switch optimization that can allow an efficient balance of exploration and exploitation mechanism for complex tasks by constraining policy parameters to parameters obtained as a result of this improved optimization. This research study also includes theoretical work on the uniqueness of control parameter estimates and also leverages a structure of the likelihood function which has an established theoretical guarantees. Furthermore, a theoretical result is also explored that bridge the gap between the proposed optimization objective function and existing information theory (relative entropy) and optimal control dualities.

📄 PDF Abstract BibTeX arXiv:2312.14463

Code (0)

등록된 구현이 없습니다.

Tasks

Model-based Reinforcement Learningreinforcement-learning

Difference of Convex Functions Programming for Reinforcement Learning

2014-12-01 · NeurIPS 2014 12 · Bilal Piot, Matthieu Geist, Olivier Pietquin

Large Markov Decision Processes (MDPs) are usually solved using Approximate Dynamic Programming (ADP) methods such as Approximate Value Iteration (AVI) or Approximate Policy Iteration (API). The main contribution of this…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Online Reinforcement Learning Control by Direct Heuristic Dynamic Programming: from Time-Driven to Event-Driven

2020-06-16 · Qingtao Zhao, Jennie Si, Jian Sun

In this paper time-driven learning refers to the machine learning method that updates parameters in a prediction model continuously as new data arrives. Among existing approximate dynamic programming (ADP) and reinforcem…

Reinforcement Learning (RL)

Model-Free Incremental Adaptive Dynamic Programming Based Approximate Robust Optimal Regulation

2021-05-04 · Cong Li, Yongchao Wang, Fangzhou Liu, Qingchen Liu 외

This paper presents a new formulation for model-free robust optimal regulation of continuous-time nonlinear systems. The proposed reinforcement learning based approach, referred to as incremental adaptive dynamic program…

Sub-optimal Tracking in Switched Systems with Controlled Subsystems and Fixed-mode Sequence using Approximate Dynamic Programming

2019-07-27

Optimal tracking in switched systems with controlled subsystem and Discrete-time (DT) dynamics is investigated. A feedback control policy is generated such that a) the system tracks the desired reference signal, and b) t…

Approximate Dynamic Programming For Linear Systems with State and Input Constraints

2019-06-26 · Ankush Chakrabarty, Rien Quirynen, Claus Danielson, Weinan Gao

Enforcing state and input constraints during reinforcement learning (RL) in continuous state spaces is an open but crucial problem which remains a roadblock to using RL in safety-critical applications. This paper leverag…