Reasoning Under Pressure: How do Training Incentives Influence Chain-of-Thought Monitorability?
AI systems that output their reasoning in natural language offer an opportunity for safety -- we can \emph{monitor} their chain of thought (CoT) for undesirable reasoning, such as the pursuit of harmful objectives. However, the extent to which CoT faithfully reflects the underlying reasoning process, and hence the extent to which it can be usefully monitored, may be influenced by certain aspects of training. We investigate how different \emph{training incentives}, applied to a reasoning model, affect its monitorability. We introduce a novel methodology for measuring monitorability according to whether a monitor can predict a key latent variable using the model's reasoning. When controlling for accuracy, we do not find evidence for consistent effects from commonly used incentives (length penalties and KL regularisation), but we find that adversarial optimisation (penalising monitor accuracy) degrades monitor performance, while direct optimisation for monitorability does not reliably lead to improvements. Our code is available at https://github.com/QiyaoWei/reasoning-under-pressure.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
How Performance Pressure Influences AI-Assisted Decision Making
Many domains now employ AI-based decision-making aids, and although the potential for AI systems to assist with decision making is much discussed, human-AI collaboration often underperforms due to factors such as (mis)tr…
Decision MakingUnderstanding Agent Incentives using Causal Influence Diagrams. Part I: Single Action Settings
Agents are systems that optimize an objective function in an environment. Together, the goal and the environment induce secondary objectives, incentives. Modeling the agent-environment interaction using causal influence …
Reinforcement LearningReinforcement Learning (RL)Effects of sparse rewards of different magnitudes in the speed of learning of model-based actor critic methods
Actor critic methods with sparse rewards in model-based deep reinforcement learning typically require a deterministic binary reward function that reflects only two possible outcomes: if, for each step, the goal has been …
Deep Reinforcement LearningMuJoCoReinforcement LearningvalidThe Energy Society: A Simulation Environment for Studying Agent Cooperation under Survival Pressure
LLM-based agents are increasingly deployed in multi-agent environments whose incentives can shape their behavior. We introduce The Energy Society, a minimal survival economy for studying how competitive and cooperative i…
The Incentives that Shape Behaviour
Which variables does an agent have an incentive to control with its decision, and which variables does it have an incentive to respond to? We formalise these incentives, and demonstrate unique graphical criteria for dete…
Fairness