TailLoR: Protecting Principal Components in Parameter-Efficient Continual Learning
Parameter-efficient finetuning methods based on spectral decomposition have enabled progress in Continual Learning. In this paper we introduce TailLoR, which utilizes the singular bases U and V of the pre-trained weights as a fixed reference frame to learn a low-rank update applied to the singular value matrix. A soft spectral penalty discourages updates aligned with dominant singular directions, reducing interference while routing fine-grained adaptation into the highly flexible, long-tail spectral coordinates.
Code (0)
등록된 구현이 없습니다.
Tasks
Continual LearningSimilar Papers 제목 키워드 기반
Continual Learning via Principal Components Projection
Continual learning in neural networks (NN) often suffers from catastrophic forgetting. That is, when learning a sequence of tasks on an NN, the learning of a new task will cause weight changes that may destroy the learne…
Continual LearningLifelong learningLeast but not Last: Fine-tuning Intermediate Principal Components for Better Performance-Forgetting Trade-Offs
Low-Rank Adaptation (LoRA) methods have emerged as crucial techniques for adapting large pre-trained models to downstream tasks under computational and memory constraints. However, they face a fundamental challenge in ba…
Continual LearningSparse principal component regression with adaptive loading
Principal component regression (PCR) is a two-stage procedure that selects some principal components and then constructs a regression model regarding them as new explanatory variables. Note that the principal components …
regressionContinual LLaVA: Continual Instruction Tuning in Large Vision-Language Models
Instruction tuning constitutes a prevalent technique for tailoring Large Vision Language Models (LVLMs) to meet individual task requirements. To date, most of the existing approaches are confined to single-task adaptatio…
ReCoLoRA: Spectrum-Aware Recursive Consolidation for Continual LLM Fine-Tuning
Parameter-efficient fine-tuning adapts a large language model to one task cheaply, but across a task sequence LoRA-style methods keep stacking low-rank updates on the same frozen weight, so each new task tends to overwri…
parameter-efficient fine-tuning