MagMax: Leveraging Model Merging for Seamless Continual Learning

2024-07-08 · Daniel Marczak, Bartłomiej Twardowski, Tomasz Trzciński, Sebastian Cygert

This paper introduces a continual learning approach named MagMax, which utilizes model merging to enable large pre-trained models to continuously learn from new data without forgetting previously acquired knowledge. Distinct from traditional continual learning methods that aim to reduce forgetting during task training, MagMax combines sequential fine-tuning with a maximum magnitude weight selection for effective knowledge integration across tasks. Our initial contribution is an extensive examination of model merging techniques, revealing that simple approaches like weight averaging and random weight selection surprisingly hold up well in various continual learning contexts. More importantly, we present MagMax, a novel model-merging strategy that enables continual learning of large pre-trained models for successive tasks. Our thorough evaluation demonstrates the superiority of MagMax in various scenarios, including class- and domain-incremental learning settings. The code is available at this URL: https://github.com/danielm1405/magmax.

📄 PDF Abstract BibTeX arXiv:2407.06322

Code (1)

danielm1405/magmax 공식 구현 pytorch

Tasks

Continual Learning Incremental Learningmodel

Tunable MAGMAX: Preference-Aware Model Merging for Continual Learning

2026-05-20 · Kei Hiroshima, Kento Uchida, Shinichi Shirakawa arxiv

Continual learning (CL) aims to train models sequentially on multiple tasks while mitigating catastrophic forgetting of previously learned knowledge. Recent advances in large pre-trained models (LPMs) and model merging t…

Continual Learning

RECALL: REpresentation-aligned Catastrophic-forgetting ALLeviation via Hierarchical Model Merging

2025-10-23 · Bowen Wang, Haiyuan Wan, Liwen Shi, Chen Yang 외 arxiv

We unveil that internal representations in large language models (LLMs) serve as reliable proxies of learned knowledge, and propose RECALL, a novel representation-aware model merging framework for continual learning with…

Continual Learning

Fast and Slow Variational Continual Learning

2026-06-22 · Subarnaduti Paul, Yohan Jung, Mohammad Emtiyaz Khan, Siddharth Swaroop 외 arxiv

Continual learning remains a major challenge for modern deep networks, partly because commonly used optimizers lack inherent mechanisms for continual adaptation. One such natural mechanism is fast and slow adaptation to …

Incremental Learning Continual Learning

MergeSlide: Continual Model Merging and Task-to-Class Prompt-Aligned Inference for Lifelong Learning on Whole Slide Images

2025-11-17 · Doanh C. Bui, Ba Hung Ngo, Hoai Luan Pham, Khang Nguyen 외 arxiv

Lifelong learning on Whole Slide Images (WSIs) aims to train or fine-tune a unified model sequentially on cancer-related tasks, reducing the resources and effort required for data transfer and processing, especially give…

class-incremental learning Continual Learning

Continual Event Extraction with Semantic Confusion Rectification

2023-10-24 · Zitao Wang, Xinyi Wang, Wei Hu

We study continual event extraction, which aims to extract incessantly emerging event information while avoiding forgetting. We observe that the semantic confusion on event types stems from the annotations of the same te…

Event Extraction Sentence