When Is Rank-1 Enough? Geometry-Guided Initialization for Parameter-Efficient Fine-Tuning

2026-02-02 · Haoran Zhao, Soyeon Caren Han, Eduard Hovy arxiv

Parameter-efficient fine-tuning (PEFT) is a standard way to adapt multimodal large language models, yet extremely low-rank settings -- especially rank-1 LoRA -- are often unstable. We show that this instability is not solely due to limited capacity: in the rank-1 regime, optimization is highly sensitive to the update direction. Concretely, pretrained vision and text features form mismatched anisotropic regions, yielding a dominant "gap" direction that acts like a translation component and disproportionately steers early gradients under rank-1 constraints. Analyzing pretrained representations, we identify a modality-gap axis that dominates early gradient flow, while a random rank-1 initialization is unlikely to align with it, leading to weak gradients and training collapse. We propose Gap-Init, a geometry-aware initialization that aligns the rank-1 LoRA direction with an estimated modality-gap vector from a small calibration set, while keeping the initial LoRA update zero. Across multiple vision-language tasks and backbones, Gap-Init consistently stabilizes rank-1 training and can match or outperform strong rank-8 baselines. Our results suggest that at the extreme low-rank limit, initial alignment can matter as much as rank itself.

📄 PDF Abstract BibTeX arXiv:2602.01522

Code (0)

등록된 구현이 없습니다.

Tasks

parameter-efficient fine-tuning

Geometry-Preserving Orthonormal Initialization for Low-Rank Adaptation in RLVR

2026-06-30 · Ruijia Zhang, Jiacheng Zhu, Hanqing Zhu, Laixi Shi arxiv

Low-rank adaptation (LoRA) and its variants enable parameter-efficient fine-tuning of large language models under the supervised fine-tuning (SFT) paradigm. However, their efficacy and behavior under Reinforcement learni…

parameter-efficient fine-tuning Reinforcement Learning Mathematical Reasoning

Learning in the Fisher Subspace: A Guided Initialization for LoRA Fine-Tuning

2026-05-01 · Zhi-Quan Feng, Ying-Jia Lin, Hung-Yu Kao arxiv

LoRA adapts large language models (LLMs) by restricting updates to low-rank subspaces of pre-trained weights. While this substantially reduces training cost, the effectiveness of adaptation critically depends on which su…

Implicit Bias in Leaky ReLU Networks Trained on High-Dimensional Data

2022-10-13 · Spencer Frei, Gal Vardi, Peter L. Bartlett, Nathan Srebro 외

The implicit biases of gradient-based optimization algorithms are conjectured to be a major factor in the success of modern deep learning. In this work, we investigate the implicit bias of gradient flow and gradient desc…

Vocal Bursts Intensity Prediction

IndoorGS: Geometric Cues Guided Gaussian Splatting for Indoor Scene Reconstruction

2025-01-01 · CVPR 2025 1 · Cong Ruan, Yuesong Wang, Tao Guan, Bin Zhang 외

3D Gaussian Splatting (3DGS) has shown impressive performance in scene reconstruction, offering high rendering quality and rapid rendering speed with short training time. However, it often yields unsatisfactory resul…

3DGS Indoor Scene Reconstruction

PanoImager: Geometry-Guided Novel View Synthesis and Reconstruction from Sparse Panoramic Views

2026-06-25 · Zhisong Xu, Takeshi Oishi arxiv

Panoramic sensing offers wide field-of-view coverage, yet 3D reconstruction from sparse panoramas remains challenging under rotation-dominant, weak-parallax motion. In such regimes, SfM/SLAM initialization is often ill-c…

Novel View Synthesis 3D Reconstruction