Mitigating Intra- and Inter-modal Forgetting in Continual Learning of Unified Multimodal Models

2025-12-02 · Xiwen Wei, Mustafa Munir, Radu Marculescu arxiv

Unified Multimodal Generative Models (UMGMs) unify visual understanding and image generation within a single autoregressive framework. However, their ability to continually learn new tasks is severely hindered by catastrophic forgetting, both within a modality (intra-modal) and across modalities (inter-modal). While intra-modal forgetting has been studied in prior continual learning (CL) work, inter-modal forgetting remains largely unexplored. In this paper, we identify and empirically validate this phenomenon in UMGMs and provide a theoretical explanation rooted in gradient conflict between modalities. To address both intra- and inter-modal forgetting, we propose Modality-Decoupled Experts (MoDE), a lightweight and scalable architecture that isolates modality-specific updates to mitigate the gradient conflict and leverages knowledge distillation to prevent catastrophic forgetting and preserve pre-trained capabilities. Unlike previous CL methods that remain modality-coupled and suffer from modality gradient conflict, MoDE explicitly decouples modalities to prevent interference. Experiments across diverse benchmarks demonstrate that MoDE significantly mitigates both inter- and intra-modal forgetting, outperforming prior CL baselines in unified multimodal generation settings. Codes will be publicly available: https://github.com/Christina200/MoDE-official.git

📄 PDF Abstract BibTeX arXiv:2512.03125

Code (0)

등록된 구현이 없습니다.

Tasks

Knowledge Distillation multimodal generation Continual Learning Image Generation

StructAlign: Structured Cross-Modal Alignment for Continual Text-to-Video Retrieval

2026-01-28 · Shaokun Wang, Weili Guan, Jizhou Han, Jianlong Wu 외 arxiv

Continual Text-to-Video Retrieval (CTVR) is a challenging multimodal continual learning setting, where models must incrementally learn new semantic categories while maintaining accurate text-video alignment for previousl…

Continual Learning Video Alignment Video Retrieval

Efficient Prompting for Continual Adaptation to Missing Modalities

2025-03-01 · Zirun Guo, Shulei Wang, Wang Lin, Weicai Yan 외

Missing modality issues are common in real-world applications, arising from factors such as equipment failures and privacy concerns. When fine-tuning pre-trained models on downstream datasets with missing modalities, per…

Continual Learning

Multimodal Continual Instruction Tuning with Dynamic Gradient Guidance

2025-11-19 · Songze Li, Mingyu Gao, Tonghua Su, Xu-Yao Zhang 외 arxiv

Multimodal continual instruction tuning enables multimodal large language models to sequentially adapt to new tasks while building upon previously acquired knowledge. However, this continual learning paradigm faces the s…

Continual Learning

CLiMB: A Continual Learning Benchmark for Vision-and-Language Tasks

2022-06-18 · Tejas Srinivasan, Ting-Yun Chang, Leticia Leonor Pinto Alva, Georgios Chochlakis 외

Current state-of-the-art vision-and-language models are evaluated on tasks either individually or in a multi-task setting, overlooking the challenges of continually learning (CL) tasks as they arrive. Existing CL benchma…

Continual Learning Transfer Learning

Beyond Unimodal Learning: The Importance of Integrating Multiple Modalities for Lifelong Learning

2024-05-04 · Fahad Sarfraz, Bahram Zonooz, Elahe Arani

While humans excel at continual learning (CL), deep neural networks (DNNs) exhibit catastrophic forgetting. A salient feature of the brain that allows effective CL is that it utilizes multiple modalities for learning and…

Continual Learning Lifelong learning