FDA-Opt: Communication-Efficient Federated Fine-Tuning of Language Models

2025-05-07 · Michail Theologitis, Vasilis Samoladas, Antonios Deligiannakis

Federated Learning (FL) enables the utilization of vast, previously inaccessible data sources. At the same time, pre-trained Language Models (LMs) have taken the world by storm and for good reason. They exhibit remarkable emergent abilities and are readily adapted to downstream tasks. This opens one of the most exciting frontiers in FL: fine-tuning LMs. Yet, a persistent challenge in FL is the frequent, rigid communication of parameters -- a problem magnified by the sheer size of these contemporary models. The FedOpt family of algorithms has become the go-to approach for FL, relying on fixed but arbitrary intervals for model exchanges. Recently, the FDA algorithm prescribed a dynamic approach by monitoring the training progress. However, it introduced a hard-to-calibrate parameter and imposed a rigid synchronization scheme. In this work, we address these limitations by proposing the FDA-Opt family of algorithms -- a unified generalization of both FDA and FedOpt. Our experimental evaluation focuses on fine-tuning LMs on downstream NLP tasks and demonstrates that FDA-Opt outperforms FedOpt even when it is configured with hyper-parameters specifically optimized for the latter. In other words, we show that FDA-Opt is a practical, drop-in replacement for FedOpt in modern FL libraries and systems: it requires no additional configuration and delivers superior performance out of the box.

📄 PDF Abstract BibTeX arXiv:2505.04535

Code (1)

miketheologitis/FDA-Opt 공식 구현 pytorch

Tasks

Federated Learning

Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytes

2023-12-11 · Zhen Qin, Daoyuan Chen, Bingchen Qian, Bolin Ding 외

Pre-trained large language models (LLMs) need fine-tuning to improve their responsiveness to natural language instructions. Federated learning offers a way to fine-tune LLMs using the abundant data on end devices without…

Federated Learning Large Language Model parameter-efficient fine-tuning

One Communication Round is All It Needs for Federated Fine-Tuning Foundation Models

2024-12-05 · Ziyao Wang, Bowei Tian, Yexiao He, Zheyu Shen 외

The recent advancement of large foundation models (FMs) has increased the demand for fine-tuning these models on large-scale and cross-domain datasets. To address this, federated fine-tuning has emerged as a solution, al…

All Image Generation Text Generation Text to Image Generation+1

FedKRSO: Communication and Memory Efficient Federated Fine-Tuning of Large Language Models

2026-02-03 · Guohao Yang, Tongle Wu, Yuanxiong Guo, Ying Sun 외 arxiv

Fine-tuning is essential to adapt general-purpose large language models (LLMs) to domain-specific tasks. As a privacy-preserving framework to leverage decentralized data for collaborative model training, Federated Learni…

parameter-efficient fine-tuning Federated Learning

Fed-SB: A Silver Bullet for Extreme Communication Efficiency and Performance in (Private) Federated LoRA Fine-Tuning

2025-02-21 · Raghav Singhal, Kaustubh Ponkshe, Rohit Vartak, Lav R. Varshney 외

Low-Rank Adaptation (LoRA) has become ubiquitous for efficiently fine-tuning foundation models. However, federated fine-tuning using LoRA is challenging due to suboptimal updates arising from traditional federated averag…

Arithmetic Reasoning

FedSRD: Sparsify-Reconstruct-Decompose for Communication-Efficient Federated Large Language Models Fine-Tuning

2025-10-06 · Guochen Yan, Luyuan Xie, Qingni Shen, Yuejian Fang 외 arxiv

The current paradigm of training large language models (LLMs) on public available Web data is becoming unsustainable as high-quality data sources in specialized domains near exhaustion. Federated Learning (FL) emerges as…