FuRA: Full-Rank Parameter-Efficient Fine-Tuning with Spectral Preconditioning

2026-05-19 · Yequan Zhao, Ruijie Zhang, Liyan Tan, Niall Moran, Tong Qin, Zheng Zhang arxiv

Both full fine-tuning (Full FT) and parameter-efficient fine-tuning methods such as LoRA introduce weight updates without accounting for the spectral structure established during pretraining. As a result, noisy gradients from limited fine-tuning data can perturb robust pretrained features. We identify spectral preconditioning as the missing ingredient: reparameterizing each weight matrix through its full-rank singular value decomposition (SVD) and freezing one singular basis constrains updates to the pretrained column space, yielding a preconditioned optimization scheme that outperforms unconstrained Full FT at the same trainable parameter count. Building on this insight, we propose FuRA (Full-Rank Adaptation), an efficient full-rank adaptation framework based on a block tensor-train factorization W = LSR, where the large core L is fixed to the pretrained block-wise SVD basis, while only the compact core R and the block-wise singular values S are optimized. This design simultaneously provides full-rank spectral preconditioning, preserves full-rank update expressivity, and achieves parameter, memory, and step-time efficiency comparable to LoRA. FuRA consistently outperforms Full FT across multiple settings, including LLM fine-tuning (+1.37 on LLaMA-3-8B commonsense reasoning), LLM reinforcement learning for mathematical reasoning, and visual instruction tuning for VLMs. Furthermore, the 4-bit quantized variant, QFuRA, also surpasses QLoRA. Code is available at https://github.com/olokevin/FuRA-NIPS

📄 PDF Abstract BibTeX arXiv:2605.22869

Code (0)

등록된 구현이 없습니다.

Tasks

parameter-efficient fine-tuning Reinforcement Learning Mathematical Reasoning

Safurai 001: New Qualitative Approach for Code LLM Evaluation

2023-09-20 · Davide Cifarelli, Leonardo Boiardi, Alessandro Puppo

This paper presents Safurai-001, a new Large Language Model (LLM) with significant potential in the domain of coding assistance. Driven by recent advancements in coding LLMs, Safurai-001 competes in performance with the …

Language Modeling Language Modelling Large Language Model Prompt Engineering

Safurai-Csharp: Harnessing Synthetic Data to improve language-specific Code LLM

2023-11-06 · Davide Cifarelli, Leonardo Boiardi, Alessandro Puppo, Leon Jovanovic

This paper introduces Safurai-Csharp, an open-source model designed to specialize in the generation, completion, and debugging of C# code. Safurai-Csharp is built upon the novel CodeLlama 34B model and leverages the Evol…

C++ code

DiffuRank: Effective Document Reranking with Diffusion Language Models

2026-02-13 · Qi Liu, Kun Ai, Jiaxin Mao, Yanzhao Zhang 외 arxiv

Recent advances in large language models (LLMs) have inspired new paradigms for document reranking. While this paradigm better exploits the reasoning and contextual understanding capabilities of LLMs, most existing LLM-b…

LoFT: Low-Rank Adaptation That Behaves Like Full Fine-Tuning

2025-05-27 · Nurbek Tastan, Stefanos Laskaridis, Martin Takac, Karthik Nandakumar 외

Large pre-trained models are commonly adapted to downstream tasks using parameter-efficient fine-tuning methods such as Low-Rank Adaptation (LoRA), which injects small trainable low-rank matrices instead of updating all …

parameter-efficient fine-tuning

Parameter-Efficient Fine-Tuning for Medical Text Summarization: A Comparative Study of Lora, Prompt Tuning, and Full Fine-Tuning

2026-03-23 · Ulugbek Shernazarov, Rostislav Svitsov, Bin Shi arxiv

Fine-tuning large language models for domain-specific tasks such as medical text summarization demands substantial computational resources. Parameter-efficient fine-tuning (PEFT) methods offer promising alternatives by u…

parameter-efficient fine-tuning Text Summarization