Avoiding Wireheading with Value Reinforcement Learning

2016-05-10 · Tom Everitt, Marcus Hutter

How can we design good goals for arbitrarily intelligent agents? Reinforcement learning (RL) is a natural approach. Unfortunately, RL does not work well for generally intelligent agents, as RL agents are incentivised to shortcut the reward sensor for maximum reward -- the so-called wireheading problem. In this paper we suggest an alternative to RL called value reinforcement learning (VRL). In VRL, agents use the reward signal to learn a utility function. The VRL setup allows us to remove the incentive to wirehead by placing a constraint on the agent's actions. The constraint is defined in terms of the agent's belief distributions, and does not require an explicit specification of which actions constitute wireheading.

📄 PDF Abstract BibTeX arXiv:1605.03143

Code (0)

등록된 구현이 없습니다.

Tasks

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Emergence of Addictive Behaviors in Reinforcement Learning Agents

2018-11-14 · Vahid Behzadan, Roman V. Yampolskiy, Arslan Munir

This paper presents a novel approach to the technical analysis of wireheading in intelligent agents. Inspired by the natural analogues of wireheading and their prevalent manifestations, we propose the modeling of such ph…

Q-Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Categorizing Wireheading in Partially Embedded Agents

2019-06-21 · Arushi Majha, Sayan Sarkar, Davide Zagami

$\textit{Embedded agents}$ are not explicitly separated from their environment, lacking clear I/O channels. Such agents can reason about and modify their internal parts, which they are incentivized to shortcut or $\texti…

Does Self-Evaluation Enable Wireheading in Language Models?

2025-11-28 · David Demitri Africa, Hans Ethan Ting arxiv

Self-evaluation is increasingly central to language model training, underpinning techniques from Constitutional AI to self-refinement. We investigate whether coupling self-evaluation to reward signals creates incentives …

Pessimistic Auxiliary Policy for Offline Reinforcement Learning

2026-02-27 · Fan Zhang, Baoru Huang, Xin Zhang arxiv

Offline reinforcement learning aims to learn an agent from pre-collected datasets, avoiding unsafe and inefficient real-time interaction. However, inevitable access to out-ofdistribution actions during the learning proce…

Reinforcement Learning Offline RL

Value function interference and greedy action selection in value-based multi-objective reinforcement learning

2024-02-09 · Peter Vamplew, Cameron Foale, Richard Dazeley

Multi-objective reinforcement learning (MORL) algorithms extend conventional reinforcement learning (RL) to the more general case of problems with multiple, conflicting objectives, represented by vector-valued rewards. W…

Multi-Objective Reinforcement Learning Q-Learning reinforcement-learning Reinforcement Learning+1