Shared LoRA Subspaces for almost Strict Continual Learning

2026-02-05 · Prakhar Kaushik, Ankit Vaidya, Shravan Chaudhari, Rama Chellappa, Alan Yuille arxiv

Adapting large pretrained models to new tasks efficiently and continually is crucial for real-world deployment but remains challenging due to catastrophic forgetting and the high cost of retraining. While parameter-efficient tuning methods like low rank adaptation (LoRA) reduce computational demands, they lack mechanisms for strict continual learning and knowledge integration, without relying on data replay, or multiple adapters. We propose Share, a novel approach to parameter efficient continual finetuning that learns and dynamically updates a single, shared low-rank subspace, enabling seamless adaptation across multiple tasks and modalities. Share constructs a foundational subspace that extracts core knowledge from past tasks and incrementally integrates new information by identifying essential subspace directions. Knowledge from each new task is incorporated into this evolving subspace, facilitating forward knowledge transfer, while minimizing catastrophic interference. This approach achieves up to 100x parameter reduction and 281x memory savings over traditional LoRA methods, maintaining performance comparable to jointly trained models. A single Share model can replace hundreds of task-specific LoRA adapters, supporting scalable, asynchronous continual learning. Experiments across image classification, natural language understanding, 3D pose estimation, and text-to-image generation validate its effectiveness, making Share a practical and scalable solution for lifelong learning in large-scale AI systems.

📄 PDF Abstract BibTeX arXiv:2602.06043

Code (0)

등록된 구현이 없습니다.

Tasks

Natural Language Understanding Text-to-Image Generation Image Classification Continual Learning

Geo-LoRA: Geometry-Aware Subspace Evolution for Low-Rank Adaptation in Continual Learning

2026-08-27 · Yibo Feng arxiv

Rehearsal-free class-incremental learning (CIL) with LoRA adapters remains challenging because the low-rank subspaces updated across tasks evolve without geometric control, causing unstable shared representations and rep…

class-incremental learning Continual Learning

Task-Driven Subspace Decomposition for Knowledge Sharing and Isolation in LoRA-based Continual Learning

2026-02-27 · Lingfeng He, De Cheng, Huaijie Wang, Xi Yang 외 arxiv

Continual Learning (CL) requires models to sequentially adapt to new tasks without forgetting old knowledge. Recently, Low-Rank Adaptation (LoRA), a representative Parameter-Efficient Fine-Tuning (PEFT) method, has gaine…

parameter-efficient fine-tuning Continual Learning

Dual-LoRA and Quality-Enhanced Pseudo Replay for Multimodal Continual Food Learning

2025-11-17 · Xinlan Wu, Bin Zhu, Feng Han, Pengkun Jiao 외 arxiv

Food analysis has become increasingly critical for health-related tasks such as personalized nutrition and chronic disease prevention. However, existing large multimodal models (LMMs) in food analysis suffer from catastr…

Semantic Similarity Continual Learning

BiLoRA: Almost-Orthogonal Parameter Spaces for Continual Learning

2025-01-01 · CVPR 2025 1 · Hao Zhu, Yifei Zhang, Junhao Dong, Piotr Koniusz

Continual learning requires models to learn tasks sequentially while maintaining a delicate balance between stability (retaining knowledge of previous tasks) and plasticity (adapting to new tasks). A key challenge is…

Continual Learning parameter-efficient fine-tuning

On Catastrophic Forgetting in Low-Rank Decomposition-Based Parameter-Efficient Fine-Tuning

2026-03-10 · Muhammad Ahmad, Jingjing Zheng, Yankai Cao arxiv

Parameter-efficient fine-tuning (PEFT) based on low-rank decomposition, such as LoRA, has become a standard for adapting large pretrained models. However, its behavior in sequential learning -- specifically regarding cat…