KeepLoRA: Continual Learning with Residual Gradient Adaptation

2026-01-27 · Mao-Lin Luo, Zi-Hao Zhou, Yi-Lin Zhang, Yuanyu Wan, Tong Wei, Min-Ling Zhang arxiv

Continual learning for pre-trained vision-language models requires balancing three competing objectives: retaining pre-trained knowledge, preserving knowledge from a sequence of learned tasks, and maintaining the plasticity to acquire new knowledge. This paper presents a simple but effective approach called KeepLoRA to effectively balance these objectives. We first analyze the knowledge retention mechanism within the model parameter space and find that general knowledge is mainly encoded in the principal subspace, while task-specific knowledge is encoded in the residual subspace. Motivated by this finding, KeepLoRA learns new tasks by restricting LoRA parameter updates in the residual subspace to prevent interfering with previously learned capabilities. Specifically, we infuse knowledge for a new task by projecting its gradient onto a subspace orthogonal to both the principal subspace of pre-trained model and the dominant directions of previous task features. Our theoretical and empirical analyses confirm that KeepLoRA balances the three objectives and achieves state-of-the-art performance. The implementation code is available at https://github.com/MaolinLuo/KeepLoRA.

📄 PDF Abstract BibTeX arXiv:2601.19659

Code (0)

등록된 구현이 없습니다.

Tasks

Continual Learning General Knowledge

KeepLoRA++: Continual Learning with Layer-Scaled Residual Gradient Adaptation

2026-06-15 · Mao-Lin Luo, Yi-Lin Zhang, Zi-Hao Zhou, Yankun Hong 외 arxiv

Visual Question Answering Image Classification Continual Learning

Normality-Preserving Continual Industrial Anomaly Detection via Orthogonal LoRA Banks

2026-06-01 · Weibai Fang, Haijun Che, Feiyang Ren, Qiancheng Lao arxiv

Continual industrial anomaly detection with diffusion models suffers from historical normality prior drift and catastrophic forgetting. Existing continual diffusion methods preserve previous knowledge through replay or c…

Anomaly Detection

SafeAnchor: Preventing Cumulative Safety Erosion in Continual Domain Adaptation of Large Language Models

2026-04-20 · Dongxin Guo, Jikun Wu, Siu Ming Yiu arxiv

Safety alignment in large language models is remarkably shallow: it is concentrated in the first few output tokens and reversible by fine-tuning on as few as 100 adversarial examples. This fragility becomes critical in r…

Domain Adaptation

iGSP:Implicit Gradient Subspace Projection for Efficient Continual Learning of Vision-Language Models

2026-05-19 · Xuezhi Cui, Dongbo Zhou, Wang Guo, Zeyuan Wang 외 arxiv

Vision-Language Models require efficient adaptation to continually emerging downstream tasks. While Parameter-Efficient Fine-Tuning mitigates catastrophic forgetting, assigning isolated modules per task leads to paramete…

parameter-efficient fine-tuning Continual Learning

Dual Low-Rank Adaptation for Continual Learning with Pre-Trained Models

2024-11-01 · Huancheng Chen, Jingtao Li, Nidham Gazagnadou, Weiming Zhuang 외

In the era of foundation models, we revisit continual learning~(CL), which aims to enable vision transformers (ViTs) to learn new tasks over time. However, as the scale of these models increases, catastrophic forgetting …

Continual Learning parameter-efficient fine-tuning