Speeding up Heterogeneous Federated Learning with Sequentially Trained Superclients
Federated Learning (FL) allows training machine learning models in privacy-constrained scenarios by enabling the cooperation of edge devices without requiring local data sharing. This approach raises several challenges due to the different statistical distribution of the local datasets and the clients' computational heterogeneity. In particular, the presence of highly non-i.i.d. data severely impairs both the performance of the trained neural network and its convergence rate, increasing the number of communication rounds requested to reach a performance comparable to that of the centralized scenario. As a solution, we propose FedSeq, a novel framework leveraging the sequential training of subgroups of heterogeneous clients, i.e. superclients, to emulate the centralized paradigm in a privacy-compliant way. Given a fixed budget of communication rounds, we show that FedSeq outperforms or match several state-of-the-art federated algorithms in terms of final performance and speed of convergence. Finally, our method can be easily integrated with other approaches available in the literature. Empirical results show that combining existing algorithms with FedSeq further improves its final performance and convergence speed. We test our method on CIFAR-10 and CIFAR-100 and prove its effectiveness in both i.i.d. and non-i.i.d. scenarios.
Code (1)
Tasks
Federated LearningImage ClassificationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Resource-constrained Federated Edge Learning with Heterogeneous Data: Formulation and Analysis
Efficient collaboration between collaborative machine learning and wireless communication technology, forming a Federated Edge Learning (FEEL), has spawned a series of next-generation intelligent applications. However, d…
Binary ClassificationEnsemble LearningMulti-class ClassificationMemory-adaptive Depth-wise Heterogenous Federated Learning
Federated learning is a promising paradigm that allows multiple clients to collaboratively train a model without sharing the local data. However, the presence of heterogeneous devices in federated learning, such as mobil…
Federated LearningFedMes: Speeding Up Federated Learning with Multiple Edge Servers
We consider federated learning with multiple wireless edge servers having their own local coverages. We focus on speeding up training in this increasingly practical setup. Our key idea is to utilize the devices located i…
Federated LearningResource-Efficient Federated Fine-Tuning Large Language Models for Heterogeneous Data
Fine-tuning large language models (LLMs) via federated learning, i.e., FedLLM, has been proposed to adapt LLMs for various downstream applications in a privacy-preserving way. To reduce the fine-tuning costs on resource-…
Federated LearningPrivacy PreservingAsyncFedED: Asynchronous Federated Learning with Euclidean Distance based Adaptive Weight Aggregation
In an asynchronous federated learning framework, the server updates the global model once it receives an update from a client instead of waiting for all the updates to arrive as in the synchronous setting. This allows he…
Federated Learning