Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resources

2024-02-18 · Jiamu Bai, Daoyuan Chen, Bingchen Qian, Liuyi Yao, Yaliang Li

Federated Learning (FL) has recently been applied to the parameter-efficient fine-tuning of Large Language Models (LLMs). While promising, it raises significant challenges due to the heterogeneous resources and data distributions of clients. This study introduces FlexLoRA, a simple yet effective aggregation scheme for LLM fine-tuning, which mitigates the ``bucket effect'' in traditional FL that restricts the potential of clients with ample resources by tying them to the capabilities of the least-resourced participants. FlexLoRA allows for dynamic adjustment of local LoRA ranks, fostering the development of a global model imbued with broader, less task-specific knowledge. By synthesizing a full-size LoRA weight from individual client contributions and employing Singular Value Decomposition (SVD) for weight redistribution, FlexLoRA fully leverages heterogeneous client resources. Involving thousands of clients performing heterogeneous NLP tasks and client resources, our experiments validate the efficacy of FlexLoRA, with the federated global model achieving consistently better improvement over SOTA FL methods in downstream NLP task performance across various heterogeneous distributions. FlexLoRA's practicality is further underscored by our theoretical analysis and its seamless integration with existing LoRA-based FL methods, offering a path toward cross-device, privacy-preserving federated tuning for LLMs.

📄 PDF Abstract BibTeX arXiv:2402.11505

Code (0)

등록된 구현이 없습니다.

Tasks

Federated Learning parameter-efficient fine-tuningPrivacy Preserving

Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytes

2023-12-11 · Zhen Qin, Daoyuan Chen, Bingchen Qian, Bolin Ding 외

Pre-trained large language models (LLMs) need fine-tuning to improve their responsiveness to natural language instructions. Federated learning offers a way to fine-tune LLMs using the abundant data on end devices without…

Federated Learning Large Language Model parameter-efficient fine-tuning

FlowerTune: A Cross-Domain Benchmark for Federated Fine-Tuning of Large Language Models

2025-06-03 · Yan Gao, Massimo Roberto Scamarcia, Javier Fernandez-Marques, Mohammad Naseri 외

Large Language Models (LLMs) have achieved state-of-the-art results across diverse domains, yet their development remains reliant on vast amounts of publicly available data, raising concerns about data scarcity and the l…

Benchmarking Domain Adaptation Federated Learning Privacy Preserving

FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations

2024-09-09 · Ziyao Wang, Zheyu Shen, Yexiao He, Guoheng Sun 외

The rapid development of Large Language Models (LLMs) has been pivotal in advancing AI, with pre-trained LLMs being adaptable to diverse downstream tasks through fine-tuning. Federated learning (FL) further enhances fine…

Federated Learning Privacy Preserving

FedHFT: Efficient Federated Finetuning with Heterogeneous Edge Clients

2025-10-15 · Fatih Ilhan, Selim Furkan Tekin, Tiansheng Huang, Gaowen Liu 외 arxiv

Fine-tuning pre-trained large language models (LLMs) has become a common practice for personalized natural language understanding (NLU) applications on downstream tasks and domain-specific datasets. However, there are tw…

Natural Language Understanding Federated Learning

DP-FedLoRA: Privacy-Enhanced Federated Fine-Tuning for On-Device Large Language Models

2025-09-11 · Honghui Xu, Shiva Shrestha, Wei Chen, Zhiyuan Li 외 arxiv

As on-device large language model (LLM) systems become increasingly prevalent, federated fine-tuning enables advanced language understanding and generation directly on edge devices; however, it also involves processing s…

Federated Learning