Bernoulli-LoRA: A Theoretical Framework for Randomized Low-Rank Adaptation

2025-08-05 · Igor Sokolov, Abdurakhmon Sadiev, Yury Demidovich, Fawaz S Al-Qahtani, Peter Richtárik arxiv

Parameter-efficient fine-tuning (PEFT) has emerged as a crucial approach for adapting large foundational models to specific tasks, particularly as model sizes continue to grow exponentially. Among PEFT methods, Low-Rank Adaptation (LoRA) (arXiv:2106.09685) stands out for its effectiveness and simplicity, expressing adaptations as a product of two low-rank matrices. While extensive empirical studies demonstrate LoRA's practical utility, theoretical understanding of such methods remains limited. Recent work on RAC-LoRA (arXiv:2410.08305) took initial steps toward rigorous analysis. In this work, we introduce Bernoulli-LoRA, a novel theoretical framework that unifies and extends existing LoRA approaches. Our method introduces a probabilistic Bernoulli mechanism for selecting which matrix to update. This approach encompasses and generalizes various existing update strategies while maintaining theoretical tractability. Under standard assumptions from non-convex optimization literature, we analyze several variants of our framework: Bernoulli-LoRA-GD, Bernoulli-LoRA-SGD, Bernoulli-LoRA-PAGE, Bernoulli-LoRA-MVR, Bernoulli-LoRA-QGD, Bernoulli-LoRA-MARINA, and Bernoulli-LoRA-EF21, establishing convergence guarantees for each variant. Additionally, we extend our analysis to convex non-smooth functions, providing convergence rates for both constant and adaptive (Polyak-type) stepsizes. Through extensive experiments on various tasks, we validate our theoretical findings and demonstrate the practical efficacy of our approach. This work is a step toward developing theoretically grounded yet practically effective PEFT methods.

📄 PDF Abstract BibTeX arXiv:2508.03820

Code (0)

등록된 구현이 없습니다.

Tasks

parameter-efficient fine-tuning

Randomized Asymmetric Chain of LoRA: The First Meaningful Theoretical Framework for Low-Rank Adaptation

2024-10-10 · Grigory Malinovsky, Umberto Michieli, Hasan Abed Al Kader Hammoud, Taha Ceritli 외

Fine-tuning has become a popular approach to adapting large foundational models to specific tasks. As the size of models and datasets grows, parameter-efficient fine-tuning techniques are increasingly important. One of t…

Federated Learning parameter-efficient fine-tuning

Guaranteed Sampling Flexibility for Low-tubal-rank Tensor Completion

2024-06-16 · Bowen Su, Juntao You, HanQin Cai, Longxiu Huang

While Bernoulli sampling is extensively studied in tensor completion, t-CUR sampling approximates low-tubal-rank tensors via lateral and horizontal subtensors. However, both methods lack sufficient flexibility for divers…

Randomized Exploration in Cooperative Multi-Agent Reinforcement Learning

2024-04-16 · Hao-Lun Hsu, Weixin Wang, Miroslav Pajic, Pan Xu

We present the first study on provably efficient randomized exploration in cooperative multi-agent reinforcement learning (MARL). We propose a unified algorithm framework for randomized exploration in parallel Markov Dec…

Federated Learning Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning+1

Geometrically Principled Randomized Optimization for Efficient LLM Training

2025-10-02 · Sahar Rajabi, Nayeema Nonta, Sirisha Rambhatla arxiv

Low-rank gradient optimization for large language models is currently divided into two categories: structured methods that rigorously identify subspaces, and randomized approaches employed primarily for computational eff…

Computational Efficiency

Perturbed-History Exploration in Stochastic Multi-Armed Bandits

2019-02-26 · Branislav Kveton, Csaba Szepesvari, Mohammad Ghavamzadeh, Craig Boutilier

We propose an online algorithm for cumulative regret minimization in a stochastic multi-armed bandit. The algorithm adds $O(t)$ i.i.d. pseudo-rewards to its history in round $t$ and then pulls the arm with the highest av…

Multi-Armed Bandits