OPLoRA: Orthogonal Projection LoRA Prevents Catastrophic Forgetting during Parameter-Efficient Fine-Tuning

2025-10-14 · Yifeng Xiong, Xiaohui Xie arxiv

Low-Rank Adaptation (LoRA) enables efficient fine-tuning of large language models but suffers from catastrophic forgetting when learned updates interfere with the dominant singular directions that encode essential pre-trained knowledge. We propose Orthogonal Projection LoRA (OPLoRA), a theoretically grounded approach that prevents this interference through double-sided orthogonal projections. By decomposing frozen weights via SVD, OPLoRA constrains LoRA updates to lie entirely within the orthogonal complement of the top-$k$ singular subspace using projections $P_L = I - U_k U_k^\top$ and $P_R = I - V_k V_k^\top$. We prove that this construction exactly preserves the top-$k$ singular triples, providing mathematical guarantees for knowledge retention. To quantify subspace interference, we introduce $ρ_k$, a metric measuring update alignment with dominant directions. Extensive experiments across commonsense reasoning, mathematics, and code generation demonstrate that OPLoRA significantly reduces forgetting while maintaining competitive task-specific performance on LLaMA-2 7B and Qwen2.5 7B, establishing orthogonal projection as an effective mechanism for knowledge preservation in parameter-efficient fine-tuning.

📄 PDF Abstract BibTeX arXiv:2510.13003

Code (0)

등록된 구현이 없습니다.

Tasks

parameter-efficient fine-tuning Code Generation

Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptation

2025-10-27 · Shiwei Li, Xiandi Luo, Haozhao Wang, Xing Tang 외 arxiv

Low-rank adaptation (LoRA) is a parameter-efficient fine-tuning (PEFT) method widely used in large language models (LLMs). LoRA essentially describes the projection of an input space into a low-dimensional output space, …

parameter-efficient fine-tuning

Faster Than SVD, Smarter Than SGD: The OPLoRA Alternating Update

2025-09-24 · Abdulla Jasem Almansoori, Maria Ivanova, Andrey Veprikov, Aleksandr Beznosikov 외 arxiv

Low-Rank Adaptation (LoRA) fine-tunes large models by learning low-rank updates on top of frozen weights, dramatically reducing trainable parameters and memory. However, there is still a gap between full training with lo…

DropLoRA: Sparse Low-Rank Adaptation for Parameter-Efficient Fine-Tuning

2025-08-24 · Haojie Zhang arxiv

LoRA-based large model parameter-efficient fine-tuning (PEFT) methods use low-rank de- composition to approximate updates to model parameters. However, compared to full- parameter fine-tuning, low-rank updates often lead…

parameter-efficient fine-tuning Mathematical Reasoning Code Generation

Normality-Preserving Continual Industrial Anomaly Detection via Orthogonal LoRA Banks

2026-06-01 · Weibai Fang, Haijun Che, Feiyang Ren, Qiancheng Lao arxiv

Continual industrial anomaly detection with diffusion models suffers from historical normality prior drift and catastrophic forgetting. Existing continual diffusion methods preserve previous knowledge through replay or c…

Anomaly Detection

GopGAN: Gradients Orthogonal Projection Generative Adversarial Network With Continual Learning

2021-07-16 · IEEE Transactions on Neural Networks and Learning Systems 2021 7 · Xiaobin Li; Weiqiang Wang

The generative adversarial networks (GANs) in continual learning suffer from catastrophic forgetting. In continual learning, GANs tend to forget about previous generation tasks and only remember the tasks they just learn…

Continual Learning Generative Adversarial Network