CLoRA: Parameter-Efficient Continual Learning with Low-Rank Adaptation

2025-07-26 · Shishir Muralidhara, Didier Stricker, René Schuster arxiv

In the past, continual learning (CL) was mostly concerned with the problem of catastrophic forgetting in neural networks, that arises when incrementally learning a sequence of tasks. Current CL methods function within the confines of limited data access, without any restrictions imposed on computational resources. However, in real-world scenarios, the latter takes precedence as deployed systems are often computationally constrained. A major drawback of most CL methods is the need to retrain the entire model for each new task. The computational demands of retraining large models can be prohibitive, limiting the applicability of CL in environments with limited resources. Through CLoRA, we explore the applicability of Low-Rank Adaptation (LoRA), a parameter-efficient fine-tuning method for class-incremental semantic segmentation. CLoRA leverages a small set of parameters of the model and uses the same set for learning across all tasks. Results demonstrate the efficacy of CLoRA, achieving performance on par with and exceeding the baseline methods. We further evaluate CLoRA using NetScore, underscoring the need to factor in resource efficiency and evaluate CL methods beyond task performance. CLoRA significantly reduces the hardware requirements for training, making it well-suited for CL in resource-constrained environments after deployment.

📄 PDF Abstract BibTeX arXiv:2507.19887

Code (0)

등록된 구현이 없습니다.

Tasks

parameter-efficient fine-tuning Semantic Segmentation Continual Learning

Collaborative Low-Rank Adaptation for Pre-Trained Vision Transformers

2025-12-31 · Zheng Liu, Jinchao Zhu, Gao Huang arxiv

Low-rank adaptation (LoRA) has achieved remarkable success in fine-tuning pre-trained vision transformers for various downstream tasks. Existing studies mainly focus on exploring more parameter-efficient strategies or mo…

Representation Learning

Controlled Low-Rank Adaptation with Subspace Regularization for Continued Training on Large Language Models

2024-10-22 · Yuheng Lu, Bingshuo Qian, Caixia Yuan, Huixing Jiang 외

Large language models (LLMs) exhibit remarkable capabilities in natural language processing but face catastrophic forgetting when learning new tasks, where adaptation to a new domain leads to a substantial decline in per…

JumpLoRA: Sparse Adapters for Continual Learning in Large Language Models

2026-04-17 · Alexandra Dragomir, Ioana Pintilie, Antonio Barbalau, Marius Dragoi 외 arxiv

Adapter-based methods have become a cost-effective approach to continual learning (CL) for Large Language Models (LLMs), by sequentially learning a low-rank update matrix for each task. To mitigate catastrophic forgettin…

Continual Learning

Robust and Efficient Fine-tuning of LLMs with Bayesian Reparameterization of Low-Rank Adaptation

2024-11-07 · Ayan Sengupta, Vaibhav Seth, Arinjay Pathak, Natraj Raman 외

Large Language Models (LLMs) are highly resource-intensive to fine-tune due to their enormous size. While low-rank adaptation is a prominent parameter-efficient fine-tuning approach, it suffers from sensitivity to hyperp…

Natural Language Understanding parameter-efficient fine-tuning parameter estimation

Distillation-Supervised Convolutional Low-Rank Adaptation for Efficient Image Super-Resolution

2025-04-15 · Xinning Chai, Yao Zhang, Yuxuan Zhang, Zhengxue Cheng 외

Convolutional neural networks (CNNs) have been widely used in efficient image super-resolution. However, for CNN-based methods, performance gains often require deeper networks and larger feature maps, which increase comp…

Image Super-Resolution Knowledge Distillation SSIM Super-Resolution