CoinRun: Solving Goal Misgeneralisation

2023-09-28 · Stuart Armstrong, Alexandre Maranhão, Oliver Daniels-Koch, Patrick Leask, Rebecca Gorman

Goal misgeneralisation is a key challenge in AI alignment -- the task of getting powerful Artificial Intelligences to align their goals with human intentions and human morality. In this paper, we show how the ACE (Algorithm for Concept Extrapolation) agent can solve one of the key standard challenges in goal misgeneralisation: the CoinRun challenge. It uses no new reward information in the new environment. This points to how autonomous agents could be trusted to act in human interests, even in novel and critical situations.

📄 PDF Abstract BibTeX arXiv:2309.16166

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

ALIGN In the ALIGN method, visual and language representations are jointly trained from noisy image alt-text data. The image and text encoders are learned via contrastive loss…

Getting By Goal Misgeneralization With a Little Help From a Mentor

2024-10-28 · Tu Trinh, Mohamad H. Danesh, Nguyen X. Khanh, Benjamin Plaut

While reinforcement learning (RL) agents often perform well during training, they can struggle with distribution shift in real-world deployments. One particularly severe risk of distribution shift is goal misgeneralizati…

Reinforcement Learning (RL)

Quantifying Generalization in Reinforcement Learning

2018-12-06 · Karl Cobbe, Oleg Klimov, Chris Hesse, Tae-hoon Kim 외

In this paper, we investigate the problem of overfitting in deep reinforcement learning. Among the most common benchmarks in RL, it is customary to use the same environments for both training and testing. This practice o…

Data Augmentation Deep Reinforcement Learning L2 Regularization reinforcement-learning+2

The Impossibility of Eliciting Latent Knowledge

2026-06-10 · Korbinian Friedl, Francis Rhys Ward, Paul Yushin Rapoport, Tom Everitt 외 arxiv

Advanced AI systems have extensive knowledge of their environments; in fact, their knowledge may (far) exceed that of their developers or users. Consequently, a desirable property for an AI system is that it is honest --…

Towards Generalizable Reinforcement Learning via Causality-Guided Self-Adaptive Representations

2024-07-30 · Yupei Yang, Biwei Huang, Fan Feng, Xinyue Wang 외

General intelligence requires quick adaption across tasks. While existing reinforcement learning (RL) methods have made progress in generalization, they typically assume only distribution changes between source and targe…

Atari Games reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1

Instance-based Generalization in Reinforcement Learning

2020-12-01 · NeurIPS 2020 12 · Martin Bertran, Natalia Martinez, Mariano Phielipp, Guillermo Sapiro

Agents trained via deep reinforcement learning (RL) routinely fail to generalize to unseen environments, even when these share the same underlying dynamics as the training levels. Understanding the generalization propert…

Deep Reinforcement Learning Generalization Bounds reinforcement-learning Reinforcement Learning+1