LoRA-DA: Data-Aware Initialization for Low-Rank Adaptation via Asymptotic Analysis

2025-10-28 · Qingyue Zhang, Chang Chu, Tianren Peng, Qi Li, Xiangyang Luo, Zhihao Jiang, Shao-Lun Huang arxiv

LoRA has become a widely adopted method for PEFT, and its initialization methods have attracted increasing attention. However, existing methods have notable limitations: many methods do not incorporate target-domain data, while gradient-based methods exploit data only at a shallow level by relying on one-step gradient decomposition. In this paper, we establish a theoretical framework for data-aware LoRA initialization. Starting from minimizing the expectation of the parameter discrepancy between the fine-tuned and target models, we derive an optimization problem with two components: a bias term, which is related to the parameter distance between the fine-tuned and target models, and is approximated using a Fisher-gradient formulation to preserve anisotropy; and a variance term, which accounts for the uncertainty introduced by sampling stochasticity through the Fisher information. Solving this problem yields an optimal initialization strategy for LoRA, based on which we develop an efficient algorithm, LoRA-DA. Empirical results across multiple benchmarks demonstrate that LoRA-DA consistently improves final accuracy over existing initialization methods. Additional studies show faster, more stable convergence, robustness across ranks, and only a small initialization overhead for LoRA-DA. The source code is available at https://github.com/zqy0126/LoRA-DA.

📄 PDF Abstract BibTeX arXiv:2510.24561

Code (0)

등록된 구현이 없습니다.

TLoRA: Task-aware Low Rank Adaptation of Large Language Models

2026-04-20 · Weicheng Lin, Yi Zhang, Jiawei Dang, Liang-Jie Zhang arxiv

Low-Rank Adaptation (LoRA) has become a widely adopted parameter-efficient fine-tuning method for large language models, with its effectiveness largely influenced by the allocation of ranks and scaling factors, as well a…

parameter-efficient fine-tuning Natural Language Understanding Code Generation

ILoRA: Federated Learning with Low-Rank Adaptation for Heterogeneous Client Aggregation

2025-11-20 · Junchao Zhou, Junkang Liu, Fanhua Shang arxiv

Federated Learning with Low-Rank Adaptation (LoRA) faces three critical challenges under client heterogeneity: (1) Initialization-Induced Instability due to random initialization misaligning client subspaces; (2) Rank In…

Federated Learning

Learning in the Fisher Subspace: A Guided Initialization for LoRA Fine-Tuning

2026-05-01 · Zhi-Quan Feng, Ying-Jia Lin, Hung-Yu Kao arxiv

LoRA adapts large language models (LLMs) by restricting updates to low-rank subspaces of pre-trained weights. While this substantially reduces training cost, the effectiveness of adaptation critically depends on which su…

Geometry-Preserving Orthonormal Initialization for Low-Rank Adaptation in RLVR

2026-06-30 · Ruijia Zhang, Jiacheng Zhu, Hanqing Zhu, Laixi Shi arxiv

Low-rank adaptation (LoRA) and its variants enable parameter-efficient fine-tuning of large language models under the supervised fine-tuning (SFT) paradigm. However, their efficacy and behavior under Reinforcement learni…

parameter-efficient fine-tuning Reinforcement Learning Mathematical Reasoning

HRP: High-Rank Preheating for Superior LoRA Initialization

2025-02-11 · Yuzhu Chen, Yingjie Wang, Shi Fu, Li Shen 외

This paper studies the crucial impact of initialization on the convergence properties of Low-Rank Adaptation (LoRA). We theoretically demonstrate that random initialization, a widely used schema, will likely lead LoRA to…