Efficient Federated Split Learning for Large Language Models over Communication Networks
Fine-tuning pre-trained large language models (LLM) in a distributed manner poses significant challenges on resource-constrained edge devices. To address this challenge, we propose FedsLLM, a novel framework that integrates split federated learning with parameter-efficient fine-tuning techniques. By leveraging model splitting and Low-Rank Adaptation (LoRA), FedsLLM reduces the computational burden on edge devices. Furthermore, the introduction of a federated server facilitates parallel training and enhances privacy. To accommodate heterogeneous communication conditions and diverse computational capabilities of edge devices, as well as the impact of LoRA rank selection on model convergence and training cost, we formulate a joint optimization problem. The formulated problem jointly optimizes subchannel allocation, power control, model splitting point selection, and LoRA rank configuration, all aimed at minimizing total training delay. An alternating optimization algorithm is developed to efficiently solve this problem and accelerate the training process. Simulation results demonstrate that the proposed FedsLLM framework achieves comparable model accuracy while significantly reducing client-side computational requirements. Furthermore, the proposed resource allocation scheme and adaptive LoRA rank selection strategy notably reduce the training latency compared to conventional approaches.
Code (0)
등록된 구현이 없습니다.
Tasks
Federated Learningparameter-efficient fine-tuningSimilar Papers 제목 키워드 기반
FedsLLM: Federated Split Learning for Large Language Models over Communication Networks
Addressing the challenges of deploying large language models in wireless communication networks, this paper combines low-rank adaptation technology (LoRA) with the splitfed learning framework to propose the federated spl…
LEMMADetailed comparison of communication efficiency of split learning and federated learning
We compare communication efficiencies of two compelling distributed machine learning approaches of split learning and federated learning. We show useful settings under which each method outperforms the other in terms of …
Federated LearningSplitFT: An Adaptive Federated Split Learning System For LLMs Fine-Tuning
Federated Split Learning has been identified as an efficient approach to address the computational resource constraints of clients in classical federated learning, while guaranteeing data privacy for distributed model tr…
Federated LearningSplitLite: Low-Rank Residual Compression for Split Learning
Federated fine-tuning of on-device large language models (LLMs) faces a significant computing burden. To overcome this limitation, split learning (SL) has emerged as a promising solution, which offloads the primary train…
SplitFedZip: Learned Compression for Data Transfer Reduction in Split-Federated Learning
Federated Learning (FL) enables multiple clients to train a collaborative model without sharing their local data. Split Learning (SL) allows a model to be trained in a split manner across different locations. Split-Feder…
Federated LearningImage SegmentationMedical Image SegmentationSemantic Segmentation