Analyzing Mitigation Strategies for Catastrophic Forgetting in End-to-End Training of Spoken Language Models

2025-05-23 · Chi-Yuan Hsiao, Ke-Han Lu, Kai-Wei Chang, Chih-Kai Yang, Wei-Chih Chen, Hung-Yi Lee

End-to-end training of Spoken Language Models (SLMs) commonly involves adapting pre-trained text-based Large Language Models (LLMs) to the speech modality through multi-stage training on diverse tasks such as ASR, TTS and spoken question answering (SQA). Although this multi-stage continual learning equips LLMs with both speech understanding and generation capabilities, the substantial differences in task and data distributions across stages can lead to catastrophic forgetting, where previously acquired knowledge is lost. This paper investigates catastrophic forgetting and evaluates three mitigation strategies-model merging, discounting the LoRA scaling factor, and experience replay to balance knowledge retention with new learning. Results show that experience replay is the most effective, with further gains achieved by combining it with other methods. These findings provide insights for developing more robust and efficient SLM training pipelines.

📄 PDF Abstract BibTeX arXiv:2505.17496

Code (0)

등록된 구현이 없습니다.

Tasks

Continual Learning Question Answering

Methods 이 논문이 사용한 방법론

Experience Replay Experience Replay is a replay memory technique used in reinforcement learning where we store the agent’s experiences at each time-step, $e\_{t} = \left(s\_{t}, a\_{t}, r\_{t},…

Foundation vs. Specialized Models: Evaluating Catastrophic Forgetting in Continual Time Series Forecasting

2025-10-01 · Nouha Karaouli, Denis Coquenet, Elisa Fromont, Martial Mermillod 외 arxiv

While Time Series Foundation Models (TSFMs) excel in zero-shot tasks, their behavior under continual fine tuning is poorly understood. We present the first systematic study of catastrophic forgetting in TSFMs (TimesFM-2.…

Time Series Forecasting Continual Learning

Provable Effects of Data Replay in Continual Learning: A Feature Learning Perspective

2026-02-02 · Meng Ding, Jinhui Xu, Kaiyi Ji arxiv

Continual learning (CL) aims to train models on a sequence of tasks while retaining performance on previously learned ones. A core challenge in this setting is catastrophic forgetting, where new learning interferes with …

Binary Classification Continual Learning

Demystifying Catastrophic Forgetting in Two-Stage Incremental Object Detector

2025-02-08 · Qirui Wu, Shizhou Zhang, De Cheng, Yinghui Xing 외

Catastrophic forgetting is a critical chanllenge for incremental object detection (IOD). Most existing methods treat the detector monolithically, relying on instance replay or knowledge distillation without analyzing com…

Incremental Learning Knowledge Distillation object-detection Object Detection

Catastrophic Forgetting Mitigation Through Plateau Phase Activity Profiling

2025-07-11 · Idan Mashiach, Oren Glickman, Tom Tirer arxiv

Catastrophic forgetting in deep neural networks occurs when learning new tasks degrades performance on previously learned tasks due to knowledge overwriting. Among the approaches to mitigate this issue, regularization te…

Benchmarking Catastrophic Forgetting Mitigation Methods in Federated Time Series Forecasting

2025-10-24 · Khaled Hallak, Oudom Kem arxiv

Catastrophic forgetting (CF) poses a persistent challenge in continual learning (CL), especially within federated learning (FL) environments characterized by non-i.i.d. time series data. While existing research has large…

Time Series Forecasting Federated Learning Continual Learning