$α$-LoRA: Effective Fine-Tuning via Base Model Rescaling
Fine-tuning has proven to be highly effective in adapting pre-trained models to perform better on new desired tasks with minimal data samples. Among the most widely used approaches are reparameterization methods, which update a target module by augmenting its frozen weight matrix with an additional trainable weight matrix. The most prominent example is Low Rank Adaption (LoRA), which gained significant attention in recent years. In this paper, we introduce a new class of reparameterization methods for transfer learning, designed to enhance the generalization ability of fine-tuned models. We establish the effectiveness of our approach in a high-dimensional binary classification setting using tools from Random Matrix Theory, and further validate our theoretical findings through more realistic experiments, such as fine-tuning LLMs.
Code (0)
등록된 구현이 없습니다.
Tasks
Binary ClassificationTransfer LearningSimilar Papers 제목 키워드 기반
Why LoRA Fails to Forget: Regularized Low-Rank Adaptation Against Backdoors in Language Models
Low-Rank Adaptation (LoRA) is widely used for parameter-efficient fine-tuning of large language models, but it is notably ineffective at removing backdoor behaviors from poisoned pretrained models when fine-tuning on cle…
parameter-efficient fine-tuningLoRA-Muon: Spectral Steepest Descent on the Low-Rank Manifold
Low-Rank Adaptation (LoRA) significantly reduces compute and memory costs for finetuning Deep Learning models but is often harder to tune than dense training: when using factor-wise optimizers such as AdamW, it is sensit…
Detecting CSAM Text-to-Image LoRAs From Weights
Low-rank adaptation (LoRA) fine-tuning has made it cheap and easy to customize open-weight image generation models for specific tasks, including the production of child sexual abuse material (CSAM). Existing moderation r…
Image GenerationA second-order-like optimizer with adaptive gradient scaling for deep learning
In this empirical article, we introduce INNAprop, an optimization algorithm that combines the INNA method with the RMSprop adaptive gradient scaling. It leverages second-order information and rescaling while keeping the …
Image ClassificationLanguage ModellingWeight Spectra Induced Efficient Model Adaptation
Large-scale foundation models have demonstrated remarkable versatility across a wide range of downstream tasks. However, fully fine-tuning these models incurs prohibitive computational costs, motivating the development o…
modelparameter-efficient fine-tuning