Categorizing Wireheading in Partially Embedded Agents

2019-06-21 · Arushi Majha, Sayan Sarkar, Davide Zagami

$\textit{Embedded agents}$ are not explicitly separated from their environment, lacking clear I/O channels. Such agents can reason about and modify their internal parts, which they are incentivized to shortcut or $\textit{wirehead}$ in order to achieve the maximal reward. In this paper, we provide a taxonomy of ways by which wireheading can occur, followed by a definition of wirehead-vulnerable agents. Starting from the fully dualistic universal agent AIXI, we introduce a spectrum of partially embedded agents and identify wireheading opportunities that such agents can exploit, experimentally demonstrating the results with the GRL simulation platform AIXIjs. We contextualize wireheading in the broader class of all misalignment problems - where the goals of the agent conflict with the goals of the human designer - and conjecture that the only other possible type of misalignment is specification gaming. Motivated by this taxonomy, we define wirehead-vulnerable agents as embedded agents that choose to behave differently from fully dualistic agents lacking access to their internal parts.

📄 PDF Abstract BibTeX arXiv:1906.09136

Code (0)

등록된 구현이 없습니다.

Avoiding Wireheading with Value Reinforcement Learning

2016-05-10 · Tom Everitt, Marcus Hutter

How can we design good goals for arbitrarily intelligent agents? Reinforcement learning (RL) is a natural approach. Unfortunately, RL does not work well for generally intelligent agents, as RL agents are incentivised to …

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Does Self-Evaluation Enable Wireheading in Language Models?

2025-11-28 · David Demitri Africa, Hans Ethan Ting arxiv

Self-evaluation is increasingly central to language model training, underpinning techniques from Constitutional AI to self-refinement. We investigate whether coupling self-evaluation to reward signals creates incentives …

Emergence of Addictive Behaviors in Reinforcement Learning Agents

2018-11-14 · Vahid Behzadan, Roman V. Yampolskiy, Arslan Munir

This paper presents a novel approach to the technical analysis of wireheading in intelligent agents. Inspired by the natural analogues of wireheading and their prevalent manifestations, we propose the modeling of such ph…

Q-Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Multi-Agent Connected Autonomous Driving using Deep Reinforcement Learning

2019-11-11 · Praveen Palanisamy

The capability to learn and adapt to changes in the driving environment is crucial for developing autonomous driving systems that are scalable beyond geo-fenced operational design domains. Deep Reinforcement Learning (RL…

Autonomous Driving Deep Reinforcement Learning reinforcement-learning Reinforcement Learning+1

Detecting and Mitigating Reward Hacking in Reinforcement Learning Systems: A Comprehensive Empirical Study

2025-07-08 · Ibne Farabi Shihab, Sanjeda Akter, Anuj Sharma

Reward hacking in Reinforcement Learning (RL) systems poses a critical threat to the deployment of autonomous agents, where agents exploit flaws in reward functions to achieve high scores without fulfilling intended obje…

MuJoCo Recommendation Systems Reinforcement Learning (RL)