Towards Adaptive Continual Model Merging via Manifold-Aware Expert Evolution

2026-04-24 · Haiyun Qiu, Xingyu Wu, Kay Chen Tan arxiv

Continual Model Merging (CMM) sequentially integrates task-specific models into a unified architecture without intensive retraining. However, existing CMM methods are hindered by a fundamental saturation-redundancy dilemma: backbone-centric approaches face parameter saturation and representation interference within fixed capacities, whereas Mixture-of-Experts (MoE) variants resort to indiscriminate expansion, incurring expert redundancy and a routing bottleneck reliant on additional data-driven optimization. To resolve these challenges, we propose MADE-IT (Manifold-Aware Dynamic Expert Evolution and Implicit rouTing), an adaptive CMM method that orchestrates expert management and activation by grounding intrinsic expert representations in manifold geometry. We introduce a projection-based subspace affinity metric coupled with a distribution-aware adaptive threshold mechanism to guide autonomous expert evolution, harmonizing diversity with architectural parsimony. Furthermore, to bypass parameterized gating networks, we design a data-free and training-free implicit routing mechanism that activates experts via feature-subspace alignment. Extensive experiments demonstrate that MADE-IT consistently outperforms strong baselines in accuracy and robustness across long-horizon and shuffled task sequences, while significantly pruning redundant experts, particularly within generic modules and early layers.

📄 PDF Abstract BibTeX arXiv:2604.22464

Code (0)

등록된 구현이 없습니다.

CAMEx: Curvature-aware Merging of Experts

2025-02-26 · Dung V. Nguyen, Minh H. Nguyen, Luc Q. Nguyen, Rachel S. Y. Teo 외

Existing methods for merging experts during model training and fine-tuning predominantly rely on Euclidean geometry, which assumes a flat parameter space. This assumption can limit the model's generalization ability, esp…

BECAME: BayEsian Continual Learning with Adaptive Model MErging

2025-04-03 · Mei Li, Yuxiang Lu, Qinyan Dai, Suizhi Huang 외

Continual Learning (CL) strives to learn incrementally across tasks while mitigating catastrophic forgetting. A key challenge in CL is balancing stability (retaining prior knowledge) and plasticity (learning new tasks). …

Continual Learning model

MINGLE: Mixtures of Null-Space Gated Low-Rank Experts for Test-Time Continual Model Merging

2025-05-17 · Zihuan Qiu, Yi Xu, Chiyuan He, Fanman Meng 외

Continual model merging integrates independently fine-tuned models sequentially without access to original training data, providing a scalable and efficient solution to continual learning. However, current methods still …

Continual Learning Mixture-of-Experts Test-time Adaptation

Unlocking the Potential of Continual Model Merging: An ODE Perspective

2026-05-19 · Lihong Lin, Haidong Kang arxiv

Continual Model Merging (CMM) enables rapid customization of foundation models by sequentially incorporating task-adapted models without repeated retraining. However, existing merging rules usually update the deployed mo…

Revisiting Prototype Rehearsal for Exemplar-Free Continual Learning: Manifold-Aware Boundary Sampling with Adaptive Class-Balanced Loss

2026-06-04 · Hongye Xu, Bartosz Krawczyk arxiv

Exemplar-free class-incremental learning (EFCIL) aims to acquire new classes over time without storing raw data. Historically, prototype rehearsal, which samples around stored class prototypes and mixes them with current…

class-incremental learning Continual Learning