HRP: High-Rank Preheating for Superior LoRA Initialization
This paper studies the crucial impact of initialization on the convergence properties of Low-Rank Adaptation (LoRA). We theoretically demonstrate that random initialization, a widely used schema, will likely lead LoRA to random low-rank results, rather than the best low-rank result. While this issue can be mitigated by adjusting initialization towards a well-informed direction, it relies on prior knowledge of the target, which is typically unknown in real-world scenarios. To approximate this well-informed initial direction, we propose High-Rank Preheating (HRP), which fine-tunes high-rank LoRA for a few steps and uses the singular value decomposition of the preheated result as a superior initialization. HRP initialization is theory-supported to combine the convergence strengths of high-rank LoRA and the generalization strengths of low-rank LoRA. Extensive experiments demonstrate that HRP significantly enhances LoRA's effectiveness across various models and tasks, achieving performance comparable to full-parameter fine-tuning and outperforming other initialization strategies.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
ILoRA: Federated Learning with Low-Rank Adaptation for Heterogeneous Client Aggregation
Federated Learning with Low-Rank Adaptation (LoRA) faces three critical challenges under client heterogeneity: (1) Initialization-Induced Instability due to random initialization misaligning client subspaces; (2) Rank In…
Federated LearningOn the Crucial Role of Initialization for Matrix Factorization
This work revisits the classical low-rank matrix factorization problem and unveils the critical role of initialization in shaping convergence rates for such nonconvex and nonsmooth optimization. We introduce Nystrom init…
Computationally Efficient Approach for Preheating of Battery Electric Vehicles before Fast Charging in Cold Climates
This paper investigates battery preheating before fast charging, for a battery electric vehicle (BEV) driving in a cold climate. To prevent the battery from performance degradation at low temperatures, a thermal manageme…
ManagementLoRA-FAIR: Federated LoRA Fine-Tuning with Aggregation and Initialization Refinement
Foundation models (FMs) achieve strong performance across diverse tasks with task-specific fine-tuning, yet full parameter fine-tuning is often computationally prohibitive for large models. Parameter-efficient fine-tunin…
Federated Learningparameter-efficient fine-tuningPrivacy PreservingPRoLoRA: Partial Rotation Empowers More Parameter-Efficient LoRA
With the rapid scaling of large language models (LLMs), serving numerous low-rank adaptations (LoRAs) concurrently has become increasingly impractical, leading to unaffordable costs and necessitating more parameter-effic…