홈 › Papers

Mechanistic origins of catastrophic forgetting: why RL preserves circuits better than SFT?

2026-05-21 · Jeanmely Rojas Nunez, Viraj Sawant, Nathan Allen, Nomgondalai Amgalanbaatar, Yannis Zongo, Vasu Sharma, Maheep Chaudhary arxiv

Fine-tuning large language models (LLMs) frequently induces catastrophic forgetting of prior capabilities. Recent work has shown that reinforcement learning (RL) retains prior capabilities more effectively than supervised fine-tuning (SFT), attributing this to policy-gradient updates remaining closer to the base policy \cite{shenfeld2025rl}. We extend this behavioral account to the mechanistic level and ask whether RL's advantage is mirrored by stronger preservation of internal computational circuits. We introduce differential circuit vulnerability, a head-level measure of how much a circuit degrades under fine-tuning, and use it to compare RL and SFT on Qwen2.5-3B-Instruct adapted to scientific question-answering. We find a clear mechanistic trade-off: SFT adapts more rapidly to the target task but produces substantially greater circuit disruption and forgetting of prior capabilities, whereas RL preserves a larger fraction of the base circuit at the cost of slower task adaptation. These findings suggest that circuit preservation may help explain why RL is more robust to catastrophic forgetting. We released our code here: https://github.com/rl-sft-circuit-research/differential-circuit-vulnerability.

📄 PDF Abstract BibTeX arXiv:2605.28860

Code (0)

등록된 구현이 없습니다.

Tasks

Reinforcement Learning

Mechanistic Analysis of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning

2026-01-26 · Gustav Olaf Yunus Laitinen-Fredriksson Lundstrom-Imanov arxiv

Sequential fine-tuning of Large Language Models (LLMs) adaptation to target tasks often triggers catastrophic forgetting, where the acquisition of novel target skills degrades ancestral capabilities. This paper presents …

When Thinking Backfires: Mechanistic Insights Into Reasoning-Induced Misalignment

2025-08-30 · Hanqi Yan, Hainiu Xu, Siya Qi, Shu Yang 외 arxiv

With the growing accessibility and wide adoption of large language models, concerns about their safety and alignment with human values have become paramount. In this paper, we identify a concerning phenomenon: Reasoning-…

Mechanistic Data Attribution: Tracing the Training Origins of Interpretable LLM Units

2026-01-29 · Jianhui Chen, Yuzhang Luo, Liangming Pan arxiv

While Mechanistic Interpretability has identified interpretable circuits in LLMs, their causal origins in training data remain elusive. We introduce Mechanistic Data Attribution (MDA), a scalable framework that employs I…

Data Augmentation

Putting a Face to Forgetting: Continual Learning meets Mechanistic Interpretability

2026-01-29 · Sergi Masip, Gido M. van de Ven, Javier Ferrando, Tinne Tuytelaars arxiv

Catastrophic forgetting in continual learning is often measured at the performance or last-layer representation level, overlooking the underlying mechanisms. We introduce a mechanistic framework that offers a geometric i…

Continual Learning

TMS: Trajectory-Mixed Supervision for Reward-Free, On-Policy SFT

2026-02-03 · Rana Muhammad Shahroz Khan, Zijie Liu, Zhen Tan, Charles Fleming 외 arxiv

Reinforcement Learning (RL) and Supervised Fine-Tuning (SFT) are the two dominant paradigms for enhancing Large Language Model (LLM) performance on downstream tasks. While RL generally preserves broader model capabilitie…