Trial without Error: Towards Safe Reinforcement Learning via Human Intervention

2017-07-17 · William Saunders, Girish Sastry, Andreas Stuhlmueller, Owain Evans

AI systems are increasingly applied to complex tasks that involve interaction with humans. During training, such systems are potentially dangerous, as they haven't yet learned to avoid actions that could cause serious harm. How can an AI system explore and learn without making a single mistake that harms humans or otherwise causes serious damage? For model-free reinforcement learning, having a human "in the loop" and ready to intervene is currently the only way to prevent all catastrophes. We formalize human intervention for RL and show how to reduce the human labor required by training a supervised learner to imitate the human's intervention decisions. We evaluate this scheme on Atari games, with a Deep RL agent being overseen by a human for four hours. When the class of catastrophes is simple, we are able to prevent all catastrophes without affecting the agent's learning (whereas an RL baseline fails due to catastrophic forgetting). However, this scheme is less successful when catastrophes are more complex: it reduces but does not eliminate catastrophes and the supervised learner fails on adversarial examples found by the agent. Extrapolating to more challenging environments, we show that our implementation would not scale (due to the infeasible amount of human labor required). We outline extensions of the scheme that are necessary if we are to train model-free agents without a single catastrophe.

📄 PDF Abstract BibTeX arXiv:1707.05173

Code (1)

gsastry/human-rl 공식 구현

Tasks

Atari Games reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Safe Reinforcement Learning

SHaRe-RL: Structured, Interactive Reinforcement Learning for Contact-Rich Industrial Assembly Tasks

2025-09-17 · Jannick Stranghöner, Philipp Hartmann, Marco Braun, Sebastian Wrede 외 arxiv

High-mix low-volume (HMLV) industrial assembly, common in small and medium-sized enterprises (SMEs), requires the same precision, safety, and reliability as high-volume automation while remaining flexible to product vari…

Reinforcement Learning

GUARD: A Safe Reinforcement Learning Benchmark

2023-05-23 · WeiYe Zhao, Yifan Sun, Feihan Li, Rui Chen 외

Due to the trial-and-error nature, it is typically challenging to apply RL algorithms to safety-critical real-world applications, such as autonomous driving, human-robot interaction, robot manipulation, etc, where such e…

Autonomous Driving Diversity reinforcement-learning Reinforcement Learning+2

Traversing Narrow Paths: A Two-Stage Reinforcement Learning Framework for Robust and Safe Humanoid Walking

2025-08-28 · TianChen Huang, Runchen Xu, Yu Wang, Wei Gao 외 arxiv

Traversing narrow paths is challenging for humanoid robots due to the sparse and safety-critical footholds required. Purely template-based or end-to-end reinforcement learning-based methods suffer from such harsh terrain…

Reinforcement Learning

Efficient Learning of Safe Driving Policy via Human-AI Copilot Optimization

2022-02-17 · ICLR 2022 4 · Quanyi Li, Zhenghao Peng, Bolei Zhou

Human intervention is an effective way to inject human knowledge into the training loop of reinforcement learning, which can bring fast learning and ensured training safety. Given the very limited budget of human interve…

Imitation Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Learning to explore when mistakes are not allowed

2025-02-19 · Charly Pecqueux-Guézénec, Stéphane Doncieux, Nicolas Perrin-Gilbert

Goal-Conditioned Reinforcement Learning (GCRL) provides a versatile framework for developing unified controllers capable of handling wide ranges of tasks, exploring environments, and adapting behaviors. However, its reli…

Safe Exploration Safe Reinforcement Learning