Catastrophic Forgetting Mitigation Through Plateau Phase Activity Profiling
Catastrophic forgetting in deep neural networks occurs when learning new tasks degrades performance on previously learned tasks due to knowledge overwriting. Among the approaches to mitigate this issue, regularization techniques aim to identify and constrain "important" parameters to preserve previous knowledge. In the highly nonconvex optimization landscape of deep learning, we propose a novel perspective: tracking parameters during the final training plateau is more effective than monitoring them throughout the entire training process. We argue that parameters that exhibit higher activity (movement and variability) during this plateau reveal directions in the loss landscape that are relatively flat, making them suitable for adaptation to new tasks while preserving knowledge from previous ones. Our comprehensive experiments demonstrate that this approach achieves superior performance in balancing catastrophic forgetting mitigation with strong performance on newly learned tasks.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
MedPEFT-CL: Dual-Phase Parameter-Efficient Continual Learning with Medical Semantic Adapter and Bidirectional Memory Consolidation
Medical vision-language segmentation models suffer from catastrophic forgetting when adapting to new anatomical structures, requiring complete retraining that limits their clinical deployment. Although continual learning…
parameter-efficient fine-tuningSemantic SimilarityContinual LearningAnalyzing Mitigation Strategies for Catastrophic Forgetting in End-to-End Training of Spoken Language Models
End-to-end training of Spoken Language Models (SLMs) commonly involves adapting pre-trained text-based Large Language Models (LLMs) to the speech modality through multi-stage training on diverse tasks such as ASR, TTS an…
Continual LearningQuestion AnsweringContinual Deep Reinforcement Learning to Prevent Catastrophic Forgetting in Jamming Mitigation
Deep Reinforcement Learning (DRL) has been highly effective in learning from and adapting to RF environments and thus detecting and mitigating jamming effects to facilitate reliable wireless communications. However, trad…
Deep Reinforcement LearningDomain adapted machine translation: What does catastrophic forgetting forget and why?
Neural Machine Translation (NMT) models can be specialized by domain adaptation, often involving fine-tuning on a dataset of interest. This process risks catastrophic forgetting: rapid loss of generic translation quality…
Domain AdaptationMachine TranslationNMTTranslationDemystifying Catastrophic Forgetting in Two-Stage Incremental Object Detector
Catastrophic forgetting is a critical chanllenge for incremental object detection (IOD). Most existing methods treat the detector monolithically, relying on instance replay or knowledge distillation without analyzing com…
Incremental LearningKnowledge Distillationobject-detectionObject Detection