Task-Driven Subspace Decomposition for Knowledge Sharing and Isolation in LoRA-based Continual Learning

2026-02-27 · Lingfeng He, De Cheng, Huaijie Wang, Xi Yang, Nannan Wang, Xinbo Gao arxiv

Continual Learning (CL) requires models to sequentially adapt to new tasks without forgetting old knowledge. Recently, Low-Rank Adaptation (LoRA), a representative Parameter-Efficient Fine-Tuning (PEFT) method, has gained increasing attention in CL. Several LoRA-based CL methods reduce interference across tasks by separating their update spaces, typically building the new space from the estimated null space of past tasks. However, they (i) overlook task-shared directions, which suppresses knowledge transfer, and (ii) fail to capture truly effective task-specific directions since these ``null bases" of old tasks can remain nearly inactive for new task under correlated tasks. To address this, we study LoRA learning capability from a projection energy perspective, and propose Low-rank Decomposition and Adaptation (LoDA). It performs a task-driven decomposition to build general and truly task-specific LoRA subspaces by solving two energy-based objectives, decoupling directions for knowledge sharing and isolation. LoDA fixes LoRA down-projections on two subspaces and learns robust up-projections via a Gradient-Aligned Optimization (GAO) approach. After each task, before integrating the LoRA updates into the backbone, LoDA derives a closed-form recalibration for the general update, approximating a feature-level joint optimum along this task-shared direction. Experiments indicate that LoDA outperforms existing CL methods. Our code is available at https://github.com/HHHLF/LoDA_ICML2026.

📄 PDF Abstract BibTeX arXiv:2603.00191

Code (0)

등록된 구현이 없습니다.

Tasks

parameter-efficient fine-tuning Continual Learning

Optimal Estimation of Shared Singular Subspaces across Multiple Noisy Matrices

2024-11-26 · Zhengchi Ma, Rong Ma

Estimating singular subspaces from noisy matrices is a fundamental problem with wide-ranging applications across various fields. Driven by the challenges of data integration and multi-view analysis, this study focuses on…

Data Integration Denoising

Learning Koopman Eigenfunctions and Invariant Subspaces from Data: Symmetric Subspace Decomposition

2020-02-22

This paper develops data-driven methods to identify eigenfunctions of the Koopman operator associated to a dynamical system and subspaces that are invariant under the operator. We build on Extended Dynamic Mode Decomposi…

Sparse Subspace-to-Expert Sharing for Task-Agnostic Continual Learning

2026-06-05 · Fatema Siddika, Md Anwar Hossen, Tanwi Mallick, Ali Jannesari arxiv

Continual learning in Large Language Models (LLMs) is hindered by the plasticity-stability dilemma, where acquiring new capabilities often leads to catastrophic forgetting of previous knowledge. Existing methods typicall…

Continual Learning

iGSP:Implicit Gradient Subspace Projection for Efficient Continual Learning of Vision-Language Models

2026-05-19 · Xuezhi Cui, Dongbo Zhou, Wang Guo, Zeyuan Wang 외 arxiv

Vision-Language Models require efficient adaptation to continually emerging downstream tasks. While Parameter-Efficient Fine-Tuning mitigates catastrophic forgetting, assigning isolated modules per task leads to paramete…