Orthogonal Low-rank Adaptation in Lie Groups for Continual Learning of Large Language Models
Large language models (LLMs) suffer from catastrophic forgetting in sequential multi-task learning. Existing parameter regularization methods (e.g., O-LoRA, N-LoRA) mitigate interference via low-rank subspace orthogonality, but additive updates distort the intrinsic geometry of model parameters. We propose \textbf{OLieRA}, a Lie group based fine-tuning framework that preserves parameter geometry through multiplicative updates while enforcing orthogonality across task subspaces. OLieRA achieves state-of-the-art performance on the Standard CL benchmark and remains highly competitive under large task sequences. It further inherits the replay-free and task-ID free inference properties of O-LoRA, establishing a principled paradigm for continual learning in LLMs.
Code (0)
등록된 구현이 없습니다.
Tasks
Multi-Task LearningContinual LearningSimilar Papers 제목 키워드 기반
Group Orthogonal Low-Rank Adaptation for RGB-T Tracking
Parameter-efficient fine-tuning has emerged as a promising paradigm in RGB-T tracking, enabling downstream task adaptation by freezing pretrained parameters and fine-tuning only a small set of parameters. This set forms …
parameter-efficient fine-tuningRgb-T TrackingOrthogonal Subspace Learning for Language Model Continual Learning
Benefiting from massive corpora and advanced hardware, large language models (LLMs) exhibit remarkable capabilities in language understanding and generation. However, their performance degrades in scenarios where multipl…
Continual LearningLanguage ModelingLanguage ModellingmodelSubspace Geometry Governs Catastrophic Forgetting in Low-Rank Adaptation
Low-Rank Adaptation (LoRA) has emerged as a parameter-efficient approach for adapting large pre-trained models, yet its behavior under continual learning remains poorly understood. We present a geometric theory character…
parameter-efficient fine-tuningContinual LearningC-LoRA: Continual Low-Rank Adaptation for Pre-trained Models
Low-Rank Adaptation (LoRA) is an efficient fine-tuning method that has been extensively applied in areas such as natural language processing and computer vision. Existing LoRA fine-tuning approaches excel in static envir…
Continual LearningMerge before Forget: A Single LoRA Continual Learning via Continual Merging
Parameter-efficient continual learning has emerged as a promising approach for large language models (LLMs) to mitigate catastrophic forgetting while enabling adaptation to new tasks. Current Low-Rank Adaptation (LoRA) c…
Continual Learning