Forgetting of task-specific knowledge in model merging-based continual learning
This paper investigates the linear merging of models in the context of continual learning (CL). Using controlled visual cues in computer vision experiments, we demonstrate that merging largely preserves or enhances shared knowledge, while unshared task-specific knowledge rapidly degrades. We further find that merging models from an incremental training process consistently outperforms merging models trained in parallel.
Code (0)
등록된 구현이 없습니다.
Tasks
Continual LearningSimilar Papers 제목 키워드 기반
MagMax: Leveraging Model Merging for Seamless Continual Learning
This paper introduces a continual learning approach named MagMax, which utilizes model merging to enable large pre-trained models to continuously learn from new data without forgetting previously acquired knowledge. Dist…
Continual LearningIncremental LearningmodelTrain with Perturbation, Infer after Merging: A Two-Stage Framework for Continual Learning
Continual Learning (CL) aims to enable models to continuously acquire new knowledge from a sequence of tasks with avoiding the forgetting of learned information. However, existing CL methods only rely on the parameters o…
Continual Learningparameter-efficient fine-tuningRECALL: REpresentation-aligned Catastrophic-forgetting ALLeviation via Hierarchical Model Merging
We unveil that internal representations in large language models (LLMs) serve as reliable proxies of learned knowledge, and propose RECALL, a novel representation-aware model merging framework for continual learning with…
Continual LearningRegCL: Continual Adaptation of Segment Anything Model via Model Merging
To address the performance limitations of the Segment Anything Model (SAM) in specific domains, existing works primarily adopt adapter-based one-step adaptation paradigms. However, some of these methods are specific deve…
Continual LearningmodelCP-MoE: Consistency-Preserving Mixture-of-Experts for Continual Learning
Catastrophic forgetting remains a major obstacle to continual learning in large language models (LLMs) and vision--language models (VLMs). Although Mixture-of-Experts (MoE) architectures offer an efficient path to scalin…
Continual LearningVisual Reasoning