LoRA-SP: Streamlined Partial Parameter Adaptation for Resource-Efficient Fine-Tuning of Large Language Models

2024-02-28 · Yichao Wu, Yafei Xiang, Shuning Huo, Yulu Gong, Penghao Liang

In addressing the computational and memory demands of fine-tuning Large Language Models(LLMs), we propose LoRA-SP(Streamlined Partial Parameter Adaptation), a novel approach utilizing randomized half-selective parameter freezing within the Low-Rank Adaptation(LoRA)framework. This method efficiently balances pre-trained knowledge retention and adaptability for task-specific optimizations. Through a randomized mechanism, LoRA-SP determines which parameters to update or freeze, significantly reducing computational and memory requirements without compromising model performance. We evaluated LoRA-SP across several benchmark NLP tasks, demonstrating its ability to achieve competitive performance with substantially lower resource consumption compared to traditional full-parameter fine-tuning and other parameter-efficient techniques. LoRA-SP innovative approach not only facilitates the deployment of advanced NLP models in resource-limited settings but also opens new research avenues into effective and efficient model adaptation strategies.

📄 PDF Abstract BibTeX arXiv:2403.08822

Code (0)

등록된 구현이 없습니다.

PRoLoRA: Partial Rotation Empowers More Parameter-Efficient LoRA

2024-02-24 · Sheng Wang, Boyang Xue, Jiacheng Ye, Jiyue Jiang 외

With the rapid scaling of large language models (LLMs), serving numerous low-rank adaptations (LoRAs) concurrently has become increasingly impractical, leading to unaffordable costs and necessitating more parameter-effic…

Flexi-LoRA with Input-Adaptive Ranks: Efficient Finetuning for Speech and Reasoning Tasks

2026-05-03 · Zongqian Li, Yixuan Su, Han Zhou, Zihao Fu 외 arxiv

Parameter-efficient fine-tuning methods like Low-Rank Adaptation (LoRA) have become essential for deploying large language models, yet their static parameter allocation remains suboptimal for inputs of varying complexity…

parameter-efficient fine-tuning Mathematical Reasoning Question Answering

CoLoRA: Continuous low-rank adaptation for reduced implicit neural modeling of parameterized partial differential equations

2024-02-22 · Jules Berman, Benjamin Peherstorfer

This work introduces reduced models based on Continuous Low Rank Adaptation (CoLoRA) that pre-train neural networks for a given partial differential equation and then continuously adapt low-rank weights in time to rapidl…

Less is More: Resource-Efficient Low-Rank Adaptation

2025-11-30 · Chunlin Tian, Xuyang Wei, Huanrong Liu, Zhijiang Guo 외 arxiv

Low-Rank Adaptation (LoRA) is a widely adopted parameter-efficient fine-tuning (PEFT) method for Large Language Models (LLMs), but it still incurs notable overhead and suffers from parameter interference in complex datas…

parameter-efficient fine-tuning Image Generation

DiffoRA: Enabling Parameter-Efficient LLM Fine-Tuning via Differential Low-Rank Matrix Adaptation

2025-02-13 · Tangyu Jiang, Haodi Wang, Chun Yuan

The Parameter-Efficient Fine-Tuning (PEFT) methods have been extensively researched for large language models in the downstream tasks. Among all the existing approaches, the Low-Rank Adaptation (LoRA) has gained populari…

parameter-efficient fine-tuning