The AI off-switch problem as a signalling game: bounded rationality and incomparability
The off-switch problem is a critical challenge in AI control: if an AI system resists being switched off, it poses a significant risk. In this paper, we model the off-switch problem as a signalling game, where a human decision-maker communicates its preferences about some underlying decision problem to an AI agent, which then selects actions to maximise the human's utility. We assume that the human is a bounded rational agent and explore various bounded rationality mechanisms. Using real machine learning models, we reprove prior results and demonstrate that a necessary condition for an AI system to refrain from disabling its off-switch is its uncertainty about the human's utility. We also analyse how message costs influence optimal strategies and extend the analysis to scenarios involving incomparability.
Code (0)
등록된 구현이 없습니다.
Tasks
AI AgentSimilar Papers 제목 키워드 기반
Beyond Nash Equilibrium: Bounded Rationality of LLMs and humans in Strategic Decision-making
Large language models are increasingly used in strategic decision-making settings, yet evidence shows that, like humans, they often deviate from full rationality. In this study, we compare LLMs and humans using experimen…
Decision MakingGood signals gone bad: dynamic signalling with switching efforts
This paper examines signalling when the sender exerts effort and receives benefits over time. Receivers only observe a noisy public signal about the effort, which has no intrinsic value. The modelling of signalling in …
Vocal Bursts Type PredictionBounded-Rational Pursuit-Evasion Games
We present a framework that incorporates the idea of bounded rationality into dynamic stochastic pursuit-evasion games. The solution of a stochastic game is characterized, in general, by its (Nash) equilibria in feedback…
Rationality and connectivity in stochastic learning for networked coordination games
Coordination is a desirable feature in many multi-agent systems such as robotic and socioeconomic networks. We consider a task allocation problem as a binary networked coordination game over an undirected regular graph. …
Game-Theoretic Modeling of Vehicle Unprotected Left Turns Considering Drivers' Bounded Rationality
Modeling the decision-making behavior of vehicles presents unique challenges, particularly during unprotected left turns at intersections, where the uncertainty of human drivers is especially pronounced. In this context,…
Autonomous Driving