AlphaLoRA: Assigning LoRA Experts Based on Layer Training Quality

2024-10-14 · Peijun Qing, Chongyang Gao, Yefan Zhou, Xingjian Diao, Yaoqing Yang, Soroush Vosoughi

Parameter-efficient fine-tuning methods, such as Low-Rank Adaptation (LoRA), are known to enhance training efficiency in Large Language Models (LLMs). Due to the limited parameters of LoRA, recent studies seek to combine LoRA with Mixture-of-Experts (MoE) to boost performance across various tasks. However, inspired by the observed redundancy in traditional MoE structures, previous studies identify similar redundancy among LoRA experts within the MoE architecture, highlighting the necessity for non-uniform allocation of LoRA experts across different layers. In this paper, we leverage Heavy-Tailed Self-Regularization (HT-SR) Theory to design a fine-grained allocation strategy. Our analysis reveals that the number of experts per layer correlates with layer training quality, which exhibits significant variability across layers. Based on this, we introduce AlphaLoRA, a theoretically principled and training-free method for allocating LoRA experts to further mitigate redundancy. Experiments on three models across ten language processing and reasoning benchmarks demonstrate that AlphaLoRA achieves comparable or superior performance over all baselines. Our code is available at https://github.com/morelife2017/alphalora.

📄 PDF Abstract BibTeX arXiv:2410.10054

Code (1)

morelife2017/alphalora 공식 구현 pytorch

Tasks

Mixture-of-Expertsparameter-efficient fine-tuning

Methods 이 논문이 사용한 방법론

MoE 설명 없음

GuiLoMo: Allocating Expert Number and Rank for LoRA-MoE via Bilevel Optimization with GuidedSelection Vectors

2025-06-17 · Hengyuan Zhang, Xinrong Chen, Yingmin Qiu, Xiao Liang 외

Parameter-efficient fine-tuning (PEFT) methods, particularly Low-Rank Adaptation (LoRA), offer an efficient way to adapt large language models with reduced computational costs. However, their performance is limited by th…

Bilevel Optimization Mixture-of-Experts parameter-efficient fine-tuning

Diversified Dynamic Routing for Vision Tasks

2022-09-26 · Botos Csaba, Adel Bibi, Yanwei Li, Philip Torr 외

Deep learning models for vision tasks are trained on large datasets under the assumption that there exists a universal representation that can be used to make predictions for all samples. Whereas high complexity models a…

Instance Segmentation Mixture-of-Experts object-detection Object Detection+1

MoE-Sieve: Routing-Guided LoRA for Efficient MoE Fine-Tuning

2026-03-25 · Andrea Manzoni arxiv

Standard LoRA fine-tuning of Mixture-of-Experts (MoE) models applies adapters to every expert, yet our profiling shows that per-layer expert routing is highly skewed: a small subset of experts handles most tokens in each…

Higher Layers Need More LoRA Experts

2024-02-13 · Chongyang Gao, Kezhen Chen, Jinmeng Rao, Baochen Sun 외

Parameter-efficient tuning (PEFT) techniques like low-rank adaptation (LoRA) offer training efficiency on Large Language Models, but their impact on model performance remains limited. Recent efforts integrate LoRA and Mi…

Mixture-of-Experts

HELLoRA: Hot Experts Layer-Level Low-Rank Adaptation for Mixture-of-Experts Models

2026-05-11 · Jia Wei, Zhonghao Zhang, Ping Chen, Qianyang li 외 arxiv

Low-Rank Adaptation (LoRA) dominates parameter-efficient fine-tuning of large language models, yet most variants target dense architectures. Mixture-of-Experts (MoE) models scale parameters at near-constant per-token com…

parameter-efficient fine-tuning Mathematical Reasoning Code Generation