Energy-Structured Low-Rank Adaptation for Continual Learning
While orthogonal subspace methods try to mitigate task interference in Continual Learning (CL), they often suffer from energy diffusion across the basis, hindering knowledge compaction and exhausting capacity for future tasks. We observe that output feature drift induced by parameter updates is inherently low-rank, and theoretically prove that preserving parameters along the principal directions of this drift minimizes the output reconstruction error. Motivated by this, we propose \textbf{E}nergy-Concentrated and \textbf{E}nergy-Ordered \textbf{Lo}w-\textbf{R}ank \textbf{A}daptation (E$^2$-LoRA). By explicitly ordering and concentrating knowledge into leading ranks, E$^2$-LoRA frees capacity for subsequent tasks. Furthermore, we design a dynamic rank allocation strategy to balance stability and plasticity by jointly optimizing energy retention and model plasticity. Extensive experiments across multiple benchmarks demonstrate that E$^2$-LoRA achieves state-of-the-art performance. Code is available at https://github.com/kiddo127/E2-LoRA.
Code (0)
등록된 구현이 없습니다.
Tasks
Continual LearningSimilar Papers 제목 키워드 기반
Spectral Imbalance Causes Forgetting in Low-Rank Continual Adaptation
Parameter-efficient continual learning aims to adapt pre-trained models to sequential tasks without forgetting previously acquired knowledge. Most existing approaches treat continual learning as avoiding interference wit…
Continual LearningParameter Efficient Continual Learning for Sparse Event-Based Transformers
Robotic and edge intelligence systems operate in dynamic environments where data arrives continuously, requiring models to adapt while preserving previously learned knowledge under strict memory and energy constraints. W…
parameter-efficient fine-tuningclass-incremental learningContinual LearningEvent-based visionContinual Fine-Tuning of Large Language Models via Program Memory
Parameter-Efficient Fine-Tuning (PEFT), particularly Low-Rank Adaptation (LoRA), has become a standard approach for adapting Large Language Models (LLMs) under limited compute. However, in continual settings where models…
parameter-efficient fine-tuningViDA: Homeostatic Visual Domain Adapter for Continual Test Time Adaptation
Since real-world machine systems are running in non-stationary environments, Continual Test-Time Adaptation (CTTA) task is proposed to adapt the pre-trained model to continually changing target domains. Recently, existin…
Test-time AdaptationC-LoRA: Continual Low-Rank Adaptation for Pre-trained Models
Low-Rank Adaptation (LoRA) is an efficient fine-tuning method that has been extensively applied in areas such as natural language processing and computer vision. Existing LoRA fine-tuning approaches excel in static envir…
Continual Learning