Small Vectors, Big Effects: A Mechanistic Study of RL-Induced Reasoning via Steering Vectors
The mechanisms by which reasoning training reshapes LLMs' internal computations remain unclear. We study lightweight steering vectors inserted into the base model's residual stream and trained with a reinforcement-learning objective. These vectors explain a large portion of full fine-tuning performance increase while preserving the interpretability of small, additive interventions. We find that (i) the last-layer steering vector acts like a token-substitution bias concentrated on the first generated token, consistently boosting tokens such as "To" and "Step"; (ii) the penultimate-layer vector leaves attention patterns largely intact and instead operates through the MLP and unembedding, preferentially up-weighting process words and structure symbols; and (iii) the steering vectors transfer to other models from the same family. Taken together, these results deepen understanding of how trained steering vectors shape computation and should inform future work in activation engineering and the study of reasoning models.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
LLM Assertiveness can be Mechanistically Decomposed into Emotional and Logical Components
Large Language Models (LLMs) often display overconfidence, presenting information with unwarranted certainty in high-stakes contexts. We investigate the internal basis of this behavior via mechanistic interpretability. U…
Understanding the Effects of Domain Finetuning on LLMs
Large Language Models (LLMs) fine-tuned for specific domains exhibit strong performance; however, the underlying mechanisms by which this fine-tuning reshapes their parametric space are not well understood. Prior works p…
Task Vectors, Learned Not Extracted: Performance Gains and Mechanistic Insight
Large Language Models (LLMs) can perform new tasks from in-context demonstrations, a phenomenon known as in-context learning (ICL). Recent work suggests that these demonstrations are compressed into task vectors (TVs), c…
Mechanisms of Prompt-Induced Hallucination in Vision-Language Models
Large vision-language models (VLMs) are highly capable, yet often hallucinate by favoring textual prompts over visual evidence. We study this failure mode in a controlled object-counting setting, where the prompt oversta…
Cooperativity transitions driven by higher-order oligomer formations in ligand-induced receptor dimerization
While cooperativity in ligand-induced receptor dimerization has been linked with receptor-receptor couplings via minimal representations of physical observables, effects arising from higher-order oligomer (e.g., trimer a…