Safe Exploration in Reinforcement Learning: A Generalized Formulation and Algorithms

2023-10-05 · NeurIPS 2023 11

Safe exploration is essential for the practical use of reinforcement learning (RL) in many real-world scenarios. In this paper, we present a generalized safe exploration (GSE) problem as a unified formulation of common safe exploration problems. We then propose a solution of the GSE problem in the form of a meta-algorithm for safe exploration, MASE, which combines an unconstrained RL algorithm with an uncertainty quantifier to guarantee safety in the current episode while properly penalizing unsafe explorations before actual safety violation to discourage them in future episodes. The advantage of MASE is that we can optimize a policy while guaranteeing with a high probability that no safety constraint will be violated under proper assumptions. Specifically, we present two variants of MASE with different constructions of the uncertainty quantifier: one based on generalized linear models with theoretical guarantees of safety and near-optimality, and another that combines a Gaussian process to ensure safety with a deep RL algorithm to maximize the reward. Finally, we demonstrate that our proposed algorithm achieves better performance than state-of-the-art algorithms on grid-world and Safety Gym benchmarks without violating any safety constraints, even during training.

📄 PDF Abstract BibTeX arXiv:2310.03225

Code (0)

등록된 구현이 없습니다.

Tasks

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Safe Exploration

Methods 이 논문이 사용한 방법론

Gaussian Process Gaussian Processes are non-parametric models for approximating functions. They rely upon a measure of similarity between points (the kernel function) to predict the value for…

A Survey of Constraint Formulations in Safe Reinforcement Learning

2024-02-03 · Akifumi Wachi, Xun Shen, Yanan Sui

Safety is critical when applying reinforcement learning (RL) to real-world problems. As a result, safe RL has emerged as a fundamental and powerful paradigm for optimizing an agent's policy while incorporating notions of…

Diversity reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+2

Safe Exploration in Markov Decision Processes

2012-05-22 · Teodor Mihai Moldovan, Pieter Abbeel

In environments with uncertain dynamics exploration is necessary to learn how to perform well. Existing reinforcement learning algorithms provide strong exploration guarantees, but they tend to rely on an ergodicity assu…

Safe Exploration

Safe Continual Reinforcement Learning Methods for Nonstationary Environments. Towards a Survey of the State of the Art

2026-01-08 · Timofey Tomashevskiy arxiv

This work provides a state-of-the-art survey of continual safe online reinforcement learning (COSRL) methods. We discuss theoretical aspects, challenges, and open questions in building continual online safe reinforcement…

Reinforcement Learning Continual Learning

Sampling-Based Safe Reinforcement Learning

2026-05-19 · Luca Vignola, Bruce D. Lee, Manish Prajapat, Manuel Wendl 외 arxiv

Safe exploration remains a fundamental challenge in reinforcement learning (RL), limiting the deployment of RL agents in the real world. We propose Sampling-Based Safe Reinforcement Learning (SBSRL), a model-based RL alg…

Reinforcement Learning Continuous Control

Safe Reinforcement Learning using Ideas from Model Predictive Control

2026-07-08 · Georg Schäfer, Jakob Rehrl, Stefan Huber, Simon Hirlaender arxiv

Reinforcement learning (RL) enables the synthesis of control policies directly from data, making it highly appealing for complex cyber-physical systems (CPSs) and robotics. A persistent challenge, however, is ensuring st…

Reinforcement Learning Active Learning