tLoRA: Efficient Multi-LoRA Training with Elastic Shared Super-Models

2026-02-06 · Kevin Li, Dibyadeep Saha, Avni Kanodia, Fan Lai arxiv

As Low-Rank Adaptation (LoRA) becomes the standard approach for efficiently fine-tuning large language models (LLMs), shared clusters increasingly execute many concurrent LoRA training jobs over the same frozen backbone. While recent advances enable batching (co-locating) multiple adapters during serving, efficient training-time co-location of heterogeneous LoRA adapters presents unique challenges. Jobs often differ in adapter rank, batch size, and resource allocation, and naïve batching can introduce synchronization stalls, communication overheads, and per-job slowdowns that are worse than executing independently. We introduce tLoRA, a framework that enables efficient batch training of multiple LoRA jobs. tLoRA fuses adapters that share the same base model into an elastic shared super-model, exploiting existing distributed training frameworks to derive parallelism plans that share resources effectively. At the kernel level, tLoRA employs a fused LoRA kernel that adaptively reconstructs low-rank computation tiles and schedules rank-aware nano-batches to maximize overlap between computation and communication across adapters. At the scheduling layer, tLoRA incorporates an online, residual-capacity-aware scheduler that adaptively groups jobs to maximize collective throughput. Evaluations using real-world cluster traces demonstrate that tLoRA improves training throughput by 1.2--1.8x, job training completion time by 2.3--5.4x, and GPU utilization by 37%.

📄 PDF Abstract BibTeX arXiv:2602.07263

Code (0)

등록된 구현이 없습니다.

SplitLoRA: A Split Parameter-Efficient Fine-Tuning Framework for Large Language Models

2024-07-01 · Zheng Lin, Xuanjie Hu, Yuxin Zhang, Zhe Chen 외

The scalability of large language models (LLMs) in handling high-complexity models and large-scale datasets has led to tremendous successes in pivotal domains. While there is an urgent need to acquire more training data …

Federated Learning parameter-efficient fine-tuning

MTLoRA: Low-Rank Adaptation Approach for Efficient Multi-Task Learning

2024-01-01 · CVPR 2024 1 · Ahmed Agiza, Marina Neseem, Sherief Reda

Adapting models pre-trained on large-scale datasets to a variety of downstream tasks is a common strategy in deep learning. Consequently parameter-efficient fine-tuning methods have emerged as a promising way to adap…

Multi-Task Learning parameter-efficient fine-tuning

MTLoRA: A Low-Rank Adaptation Approach for Efficient Multi-Task Learning

2024-03-29 · Ahmed Agiza, Marina Neseem, Sherief Reda

Adapting models pre-trained on large-scale datasets to a variety of downstream tasks is a common strategy in deep learning. Consequently, parameter-efficient fine-tuning methods have emerged as a promising way to adapt p…

Multi-Task Learning parameter-efficient fine-tuning

IntLoRA: Integral Low-rank Adaptation of Quantized Diffusion Models

2024-10-29 · Hang Guo, Yawei Li, Tao Dai, Shu-Tao Xia 외

Fine-tuning pre-trained diffusion models under limited budgets has gained great success. In particular, the recent advances that directly fine-tune the quantized weights using Low-rank Adaptation (LoRA) further reduces t…

parameter-efficient fine-tuning Quantization

TLoRA: Task-aware Low Rank Adaptation of Large Language Models

2026-04-20 · Weicheng Lin, Yi Zhang, Jiawei Dang, Liang-Jie Zhang arxiv

Low-Rank Adaptation (LoRA) has become a widely adopted parameter-efficient fine-tuning method for large language models, with its effectiveness largely influenced by the allocation of ranks and scaling factors, as well a…

parameter-efficient fine-tuning Natural Language Understanding Code Generation