Provable Safe Reinforcement Learning with Binary Feedback

2022-10-26 · Andrew Bennett, Dipendra Misra, Nathan Kallus

Safety is a crucial necessity in many applications of reinforcement learning (RL), whether robotic, automotive, or medical. Many existing approaches to safe RL rely on receiving numeric safety feedback, but in many cases this feedback can only take binary values; that is, whether an action in a given state is safe or unsafe. This is particularly true when feedback comes from human experts. We therefore consider the problem of provable safe RL when given access to an offline oracle providing binary feedback on the safety of state, action pairs. We provide a novel meta algorithm, SABRE, which can be applied to any MDP setting given access to a blackbox PAC RL algorithm for that setting. SABRE applies concepts from active learning to reinforcement learning to provably control the number of queries to the safety oracle. SABRE works by iteratively exploring the state space to find regions where the agent is currently uncertain about safety. Our main theoretical results shows that, under appropriate technical assumptions, SABRE never takes unsafe actions during training, and is guaranteed to return a near-optimal safe policy with high probability. We provide a discussion of how our meta-algorithm may be applied to various settings studied in both theoretical and empirical frameworks.

📄 PDF Abstract BibTeX arXiv:2210.14492

Code (1)

microsoft/intrepid pytorch

Tasks

Active Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Safe Reinforcement Learning

Long-term Safe Reinforcement Learning with Binary Feedback

2024-01-08 · Akifumi Wachi, Wataru Hashimoto, Kazumune Hashimoto

Safety is an indispensable requirement for applying reinforcement learning (RL) to real problems. Although there has been a surge of safe RL algorithms proposed in recent years, most existing work typically 1) relies on …

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Safe Reinforcement Learning

Safe Deep Model-Based Reinforcement Learning with Lyapunov Functions

2024-05-25 · Harry Zhang

Model-based Reinforcement Learning (MBRL) has shown many desirable properties for intelligent control tasks. However, satisfying safety and stability constraints during training and rollout remains an open question. We p…

Model-based Reinforcement Learning Model Predictive Control reinforcement-learning Reinforcement Learning

Safe Reinforcement Learning via Recovery-based Shielding with Gaussian Process Dynamics Models

2026-02-12 · Alexander W. Goodall, Francesco Belardinelli arxiv

Reinforcement learning (RL) is a powerful framework for optimal decision-making and control but often lacks provable guarantees for safety-critical applications. In this paper, we introduce a novel recovery-based shieldi…

Reinforcement Learning Continuous Control

Alignment and Safety of Diffusion Models via Reinforcement Learning and Reward Modeling: A Survey

2025-05-23 · Preeti Lamba, Kiran Ravish, Ankita Kushwaha, Pawan Kumar

Diffusion models have emerged as leading generative models for images and other modalities, but aligning their outputs with human preferences and safety constraints remains a critical challenge. This thesis proposal inve…

Active Learning Reinforcement Learning (RL)Safety Alignment

Constrained Meta Reinforcement Learning with Provable Test-Time Safety

2026-01-29 · Tingting Ni, Maryam Kamgarpour arxiv

Meta reinforcement learning (RL) allows agents to leverage experience across a distribution of tasks on which the agent can train at will, enabling faster learning of optimal policies on new test tasks. Despite its succe…

Reinforcement Learning