Constrained Policy Gradient Method for Safe and Fast Reinforcement Learning: a Neural Tangent Kernel Based Approach

2021-07-19 · Balázs Varga, Balázs Kulcsár, Morteza Haghir Chehreghani

This paper presents a constrained policy gradient algorithm. We introduce constraints for safe learning with the following steps. First, learning is slowed down (lazy learning) so that the episodic policy change can be computed with the help of the policy gradient theorem and the neural tangent kernel. Then, this enables us the evaluation of the policy at arbitrary states too. In the same spirit, learning can be guided, ensuring safety via augmenting episode batches with states where the desired action probabilities are prescribed. Finally, exogenous discounted sum of future rewards (returns) can be computed at these specific state-action pairs such that the policy network satisfies constraints. Computing the returns is based on solving a system of linear equations (equality constraints) or a constrained quadratic program (inequality constraints, regional constraints). Simulation results suggest that adding constraints (external information) to the learning can improve learning in terms of speed and transparency reasonably if constraints are appropriately selected. The efficiency of the constrained learning was demonstrated with a shallow and wide ReLU network in the Cartpole and Lunar Lander OpenAI gym environments. The main novelty of the paper is giving a practical use of the neural tangent kernel in reinforcement learning.

📄 PDF Abstract BibTeX arXiv:2107.09139

Code (1)

bva-bme/Constrained_Policy_Gradient 공식 구현 pytorch

Tasks

OpenAI Gym

Anytime Safe Reinforcement Learning

2025-04-23 · Pol Mestres, Arnau Marzabal, Jorge Cortés

This paper considers the problem of solving constrained reinforcement learning problems with anytime guarantees, meaning that the algorithmic solution returns a safe policy regardless of when it is terminated. Drawing in…

reinforcement-learning Reinforcement Learning Safe Reinforcement Learning

Model-based Constrained Reinforcement Learning using Generalized Control Barrier Function

2021-03-02 · Haitong Ma, Jianyu Chen, Shengbo Eben Li, Ziyu Lin 외

Model information can be used to predict future trajectories, so it has huge potential to avoid dangerous region when implementing reinforcement learning (RL) on real-world tasks, like autonomous driving. However, existi…

Autonomous Driving Collision Avoidance reinforcement-learning Reinforcement Learning+2

Safe Reinforcement Learning with Chance-constrained Model Predictive Control

2021-12-27 · Samuel Pfrommer, Tanmay Gautam, Alec Zhou, Somayeh Sojoudi

Real-world reinforcement learning (RL) problems often demand that agents behave safely by obeying a set of designed constraints. We address the challenge of safe RL by coupling a safety guide based on model predictive co…

Model Predictive Control reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1

Directed Policy Gradient for Safe Reinforcement Learning with Human Advice

2018-08-13 · Hélène Plisnier, Denis Steckelmacher, Tim Brys, Diederik M. Roijers 외

Many currently deployed Reinforcement Learning agents work in an environment shared with humans, be them co-workers, users or clients. It is desirable that these agents adjust to people's preferences, learn faster thanks…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Safe Reinforcement Learning

Model-Based Actor-Critic with Chance Constraint for Stochastic System

2020-12-19 · Baiyu Peng, Yao Mu, Yang Guan, Shengbo Eben Li 외

Safety is essential for reinforcement learning (RL) applied in real-world situations. Chance constraints are suitable to represent the safety requirements in stochastic systems. Previous chance-constrained RL methods usu…

Model Predictive Control Reinforcement Learning (RL)