홈 › Papers

AI Safety Gridworlds

2017-11-27 · Jan Leike, Miljan Martic, Victoria Krakovna, Pedro A. Ortega, Tom Everitt, Andrew Lefrancq, Laurent Orseau, Shane Legg

We present a suite of reinforcement learning environments illustrating various safety properties of intelligent agents. These problems include safe interruptibility, avoiding side effects, absent supervisor, reward gaming, safe exploration, as well as robustness to self-modification, distributional shift, and adversaries. To measure compliance with the intended safe behavior, we equip each environment with a performance function that is hidden from the agent. This allows us to categorize AI safety problems into robustness and specification problems, depending on whether the performance function corresponds to the observed reward function. We evaluate A2C and Rainbow, two recent deep reinforcement learning agents, on our environments and show that they are not able to solve them satisfactorily.

📄 PDF Abstract BibTeX arXiv:1711.09883

Code (2)

deepmind/ai-safety-gridworlds 공식 구현 tf

sharisun18/Absent_Supervisor_Env

Tasks

Deep Reinforcement Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Safe Exploration

Methods 이 논문이 사용한 방법론

A2C A2C, or Advantage Actor Critic, is a synchronous version of the A3C policy gradient method. As an alternative to the asynchronous…

Detecting danger in gridworlds using Gromov's Link Condition

2022-01-17 · Thomas F Burns, Robert Tang

Gridworlds have been long-utilised in AI research, particularly in reinforcement learning, as they provide simple yet scalable models for many real-world applications such as robot navigation, emergent behaviour, and ope…

Robot Navigation

Reward Hacking in Language Model Agents: Revisiting AI Safety Gridworlds

2026-06-13 · Ömer Veysel Çağatan, Xuandong Zhao arxiv

Reward hacking, where AI systems exploit misspecified objectives to achieve high reward without satisfying intended goals, remains a central challenge in AI safety. Yet most known instances have been discovered post hoc …

Reinforcement Learning

Parenting: Safe Reinforcement Learning from Human Input

2019-02-18 · Christopher Frye, Ilya Feige

Autonomous agents trained via reinforcement learning present numerous safety concerns: reward hacking, negative side effects, and unsafe exploration, among others. In the context of near-future autonomous agents, operati…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Safe Reinforcement Learning

Automata Learning meets Shielding

2022-12-04 · Martin Tappler, Stefan Pranger, Bettina Könighofer, Edi Muškardin 외

Safety is still one of the major research challenges in reinforcement learning (RL). In this paper, we address the problem of how to avoid safety violations of RL agents during exploration in probabilistic and partially …

Q-Learning Reinforcement Learning (RL)

A study of first-passage time minimization via Q-learning in heated gridworlds

2021-10-05 · M. A. Larchenko, P. Osinenko, G. Yaremenko, V. V. Palyulin

Optimization of first-passage times is required in applications ranging from nanobots navigation to market trading. In such settings, one often encounters unevenly distributed noise levels across the environment. We exte…

Q-Learning reinforcement-learning Reinforcement Learning (RL)