NET-FLEET: Achieving Linear Convergence Speedup for Fully Decentralized Federated Learning with Heterogeneous Data
Federated learning (FL) has received a surge of interest in recent years thanks to its benefits in data privacy protection, efficient communication, and parallel data processing. Also, with appropriate algorithmic designs, one could achieve the desirable linear speedup for convergence effect in FL. However, most existing works on FL are limited to systems with i.i.d. data and centralized parameter servers and results on decentralized FL with heterogeneous datasets remains limited. Moreover, whether or not the linear speedup for convergence is achievable under fully decentralized FL with data heterogeneity remains an open question. In this paper, we address these challenges by proposing a new algorithm, called NET-FLEET, for fully decentralized FL systems with data heterogeneity. The key idea of our algorithm is to enhance the local update scheme in FL (originally intended for communication efficiency) by incorporating a recursive gradient correction technique to handle heterogeneous datasets. We show that, under appropriate parameter settings, the proposed NET-FLEET algorithm achieves a linear speedup for convergence. We further conduct extensive numerical experiments to evaluate the performance of the proposed NET-FLEET algorithm and verify our theoretical findings.
Code (0)
등록된 구현이 없습니다.
Tasks
Federated LearningOpen-Ended Question AnsweringSimilar Papers 제목 키워드 기반
DePRL: Achieving Linear Convergence Speedup in Personalized Decentralized Learning with Shared Representations
Decentralized learning has emerged as an alternative method to the popular parameter-server framework which suffers from high communication burden, single-point failure and scalability issues due to the need of a central…
Learning TheoryRepresentation LearningAchieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learning
Federated learning (FL) is a distributed machine learning architecture that leverages a large number of workers to jointly learn a model with decentralized data. FL has received increasing attention in recent years thank…
Federated LearningOpen-Ended Question AnsweringAchieving Linear Speedup in Non-IID Federated Bilevel Learning
Federated bilevel optimization has received increasing attention in various emerging machine learning and communication applications. Recently, several Hessian-vector-based algorithms have been proposed to solve the fede…
Bilevel OptimizationFederated LearningAchieving Efficient Distributed Machine Learning Using a Novel Non-Linear Class of Aggregation Functions
Distributed machine learning (DML) over time-varying networks can be an enabler for emerging decentralized ML applications such as autonomous driving and drone fleeting. However, the commonly used weighted arithmetic mea…
Autonomous DrivingAchieving Linear Speedup with ProxSkip in Distributed Stochastic Optimization
The ProxSkip algorithm for distributed optimization is gaining increasing attention due to its proven benefits in accelerating communication complexity while maintaining robustness against data heterogeneity. However, ex…
Distributed OptimizationFederated LearningStochastic Optimization