Towards Safe Continuing Task Reinforcement Learning

2021-02-24 · Miguel Calvo-Fullana, Luiz F. O. Chamon, Santiago Paternain

Safety is a critical feature of controller design for physical systems. When designing control policies, several approaches to guarantee this aspect of autonomy have been proposed, such as robust controllers or control barrier functions. However, these solutions strongly rely on the model of the system being available to the designer. As a parallel development, reinforcement learning provides model-agnostic control solutions but in general, it lacks the theoretical guarantees required for safety. Recent advances show that under mild conditions, control policies can be learned via reinforcement learning, which can be guaranteed to be safe by imposing these requirements as constraints of an optimization problem. However, to transfer from learning safety to learning safely, there are two hurdles that need to be overcome: (i) it has to be possible to learn the policy without having to re-initialize the system; and (ii) the rollouts of the system need to be in themselves safe. In this paper, we tackle the first issue, proposing an algorithm capable of operating in the continuing task setting without the need of restarts. We evaluate our approach in a numerical example, which shows the capabilities of the proposed approach in learning safe policies via safe exploration.

📄 PDF Abstract BibTeX arXiv:2102.12585

Code (0)

등록된 구현이 없습니다.

Tasks

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Safe Exploration

An Empirical Study of Deep Reinforcement Learning in Continuing Tasks

2025-01-12 · Yi Wan, Dmytro Korenkevych, Zheqing Zhu

In reinforcement learning (RL), continuing tasks refer to tasks where the agent-environment interaction is ongoing and can not be broken down into episodes. These tasks are suitable when environment resets are unavailabl…

Deep Reinforcement Learning MuJoCo Q-Learning Reinforcement Learning (RL)

Discounted Reinforcement Learning Is Not an Optimization Problem

2019-10-04 · Abhishek Naik, Roshan Shariff, Niko Yasui, Hengshuai Yao 외

Discounted reinforcement learning is fundamentally incompatible with function approximation for control in continuing tasks. It is not an optimization problem in its usual formulation, so when using function approximatio…

Misconceptions reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Safe Policies for Reinforcement Learning via Primal-Dual Methods

2019-11-20 · Santiago Paternain, Miguel Calvo-Fullana, Luiz. F. O. Chamon, Alejandro Ribeiro

In this paper, we study the learning of safe policies in the setting of reinforcement learning problems. This is, we aim to control a Markov Decision Process (MDP) of which we do not know the transition probabilities, bu…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Policy Gradient Primal-Dual Method for Safe Reinforcement Learning from Human Feedback

2026-04-21 · Qiang Liu, Adrienne Kline, Ermin Wei arxiv

Safe Reinforcement Learning from Human Feedback (Safe RLHF) has recently achieved empirical success in developing helpful and harmless large language models by decoupling human preferences regarding helpfulness and harml…

Reinforcement Learning

Temporal-Logic-Based Reward Shaping for Continuing Reinforcement Learning Tasks

2020-07-03 · Yuqian Jiang, Sudarshanan Bharadwaj, Bo Wu, Rishi Shah 외

In continuing tasks, average-reward reinforcement learning may be a more appropriate problem formulation than the more common discounted reward formulation. As usual, learning an optimal policy in this setting typically …