LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits
Fine-tuning large language models (LLMs) is increasingly costly as models scale to hundreds of billions of parameters, and even parameter-efficient fine-tuning (PEFT) methods like LoRA remain resource-intensive. We introduce LowRA, the first framework to enable LoRA fine-tuning below 2 bits per parameter with minimal performance loss. LowRA optimizes fine-grained quantization - mapping, threshold selection, and precision assignment - while leveraging efficient CUDA kernels for scalable deployment. Extensive evaluations across 4 LLMs and 4 datasets show that LowRA achieves a superior performance-precision trade-off above 2 bits and remains accurate down to 1.15 bits, reducing memory usage by up to 50%. Our results highlight the potential of ultra-low-bit LoRA fine-tuning for resource-constrained environments.
Code (0)
등록된 구현이 없습니다.
Tasks
parameter-efficient fine-tuningQuantizationSimilar Papers 제목 키워드 기반
FinLoRA: Finetuning Quantized Financial Large Language Models Using Low-Rank Adaptation
Finetuned large language models (LLMs) have shown remarkable performance in financial tasks, such as sentiment analysis and information retrieval. Due to privacy concerns, finetuning and deploying Financial LLMs (FinLLMs…
GPUInformation RetrievalQuantizationRetrieval+1Enhancing Low-Rank Adaptation with Structured Nonlinear Transformations
Low-Rank Adaptation (LoRA) is a widely adopted parameter-efficient fine-tuning method for large language models. However, its linear nature limits expressiveness. We propose LoRAN, a non-linear extension of LoRA that app…
parameter-efficient fine-tuningInstruction-Based Fine-tuning of Open-Source LLMs for Predicting Customer Purchase Behaviors
In this study, the performance of various predictive models, including probabilistic baseline, CNN, LSTM, and finetuned LLMs, in forecasting merchant categories from financial transaction data have been evaluated. Utiliz…
MarketingCoLA: Cross-Modal Low-rank Adaptation for Multimodal Downstream Tasks
Foundation models have revolutionized AI, but adapting them efficiently for multimodal tasks, particularly in dual-stream architectures composed of unimodal encoders, such as DINO and BERT, remains a significant challeng…
Visual GroundingAcoustic prediction of flowrate: varying liquid jet stream onto a free surface
Information on liquid jet stream flow is crucial in many real world applications. In a large number of cases, these flows fall directly onto free surfaces (e.g. pools), creating a splash with accompanying splashing sound…