Actor-Critic Reinforcement Learning for Control with Stability Guarantee

2020-04-29 · Minghao Han, Lixian Zhang, Jun Wang, Wei Pan

Reinforcement Learning (RL) and its integration with deep learning have achieved impressive performance in various robotic control tasks, ranging from motion planning and navigation to end-to-end visual manipulation. However, stability is not guaranteed in model-free RL by solely using data. From a control-theoretic perspective, stability is the most important property for any control system, since it is closely related to safety, robustness, and reliability of robotic systems. In this paper, we propose an actor-critic RL framework for control which can guarantee closed-loop stability by employing the classic Lyapunov's method in control theory. First of all, a data-based stability theorem is proposed for stochastic nonlinear systems modeled by Markov decision process. Then we show that the stability condition could be exploited as the critic in the actor-critic RL to learn a controller/policy. At last, the effectiveness of our approach is evaluated on several well-known 3-dimensional robot control tasks and a synthetic biology gene network tracking task in three different popular physics simulation platforms. As an empirical evaluation on the advantage of stability, we show that the learned policies can enable the systems to recover to the equilibrium or way-points when interfered by uncertainties such as system parametric variations and external disturbances to a certain extent.

📄 PDF Abstract BibTeX arXiv:2004.14288

Code (1)

hithmh/Actor-critic-with-stability-guarantee 공식 구현 tf

Tasks

Motion Planning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

A note on stabilizing reinforcement learning

2021-11-24 · Pavel Osinenko, Grigory Yaremenko, Ilya Osokin

Reinforcement learning is a general methodology of adaptive optimal control that has attracted much attention in various fields ranging from video game industry to robot manipulators. Despite its remarkable performance d…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Stabilizing Off-Policy Reinforcement Learning with Conservative Policy Gradients

2019-09-25 · Chen Tessler, Nadav Merlis, Shie Mannor

In recent years, advances in deep learning have enabled the application of reinforcement learning algorithms in complex domains. However, they lack the theoretical guarantees which are present in the tabular setting and …

Deep Reinforcement Learning MuJoCo reinforcement-learning Reinforcement Learning+1

Reinforcement Learning for Distributed Transient Frequency Control with Stability and Safety Guarantees

2022-07-07 · Zhenyi Yuan, Changhong Zhao, Jorge Cortes

This paper proposes a reinforcement learning-based approach for optimal transient frequency control in power systems with stability and safety guarantees. Building on Lyapunov stability theory and safety-critical control…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Lyapunov-Based Reinforcement Learning for Decentralized Multi-Agent Control

2020-09-20 · Qingrui Zhang, Hao Dong, Wei Pan

Decentralized multi-agent control has broad applications, ranging from multi-robot cooperation to distributed sensor networks. In decentralized multi-agent control, systems are complex with unknown or highly uncertain dy…

Deep Reinforcement Learning Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning+1

Wasserstein Adaptive Value Estimation for Actor-Critic Reinforcement Learning

2025-01-17 · Ali Baheri, Zahra Shahrooei, Chirayu Salgarkar

We present Wasserstein Adaptive Value Estimation for Actor-Critic (WAVE), an approach to enhance stability in deep reinforcement learning through adaptive Wasserstein regularization. Our method addresses the inherent ins…

Computational Efficiency Deep Reinforcement Learning reinforcement-learning Reinforcement Learning