Sharpness-Aware Pretraining Mitigates Catastrophic Forgetting

2026-05-04 · Ishaan Watts, Catherine Li, Sachin Goyal, Jacob Mitchell Springer, Aditi Raghunathan arxiv

Pretraining optimizers are tuned to produce the strongest possible base model, on the assumption that a stronger starting point yields a stronger model after subsequent changes like post-training and quantization. This overlooks the geometry of the base model which controls how much of the base model's capabilities survive subsequent parameter updates. We study three pretraining optimization approaches that bias optimization toward flatter minima: Sharpness-Aware Minimization (SAM), large learning rates, and shortened learning rate annealing periods. Across model sizes ranging from 20M to 150M parameters, we find that these interventions consistently improve downstream performance after post-training on five common datasets with up to 80% less forgetting. These principles hold at scale: a short SAM mid-training phase applied to an existing OLMo-2-1B checkpoint reduces forgetting by 31% after MetaMath post-training and by 40% after 4-bit quantization.

📄 PDF Abstract BibTeX arXiv:2605.02105

Code (0)

등록된 구현이 없습니다.

(How) Learning Rates Regulate Catastrophic Overtraining

2026-04-15 · Mark Rofin, Aditya Varre, Nicolas Flammarion arxiv

Supervised fine-tuning (SFT) is a common first stage of LLM post-training, teaching the model to follow instructions and shaping its behavior as a helpful assistant. At the same time, SFT may harm the fundamental capabil…

Parameter Alignment Mitigates Catastrophic Forgetting in Multilingual Expert Language Models

2026-05-29 · Sanchit Ahuja, Terra Blevins arxiv

While continual pretraining~(CPT) is a practical way to extend large language models to new languages, naïve finetuning on targeted data erodes existing capabilities through catastrophic forgetting. Organizing training a…

Reading Comprehension Continual Pretraining Language Acquisition General Knowledge

Revisiting Catastrophic Forgetting in Large Language Model Tuning

2024-06-07 · Hongyu Li, Liang Ding, Meng Fang, DaCheng Tao

Catastrophic Forgetting (CF) means models forgetting previously acquired knowledge when learning new data. It compromises the effectiveness of large language models (LLMs) during fine-tuning, yet the underlying causes ha…

Language Modeling Language Modelling Large Language Model model

On Catastrophic Forgetting and Mode Collapse in Generative Adversarial Networks

2018-07-11 · Hoang Thanh-Tung, Truyen Tran

In this paper, we show that Generative Adversarial Networks (GANs) suffer from catastrophic forgetting even when they are trained to approximate a single target distribution. We show that GAN training is a continual lear…

Continual Learning

Effect of scale on catastrophic forgetting in neural networks

2021-09-29 · ICLR 2022 4 · Vinay Venkatesh Ramasesh, Aitor Lewkowycz, Ethan Dyer

Catastrophic forgetting presents a challenge in developing deep learning models capable of continual learning, i.e. learning tasks sequentially. Recently, both computer vision and natural-language processing have witness…