Communication-Efficient Federated Optimization over Semi-Decentralized Networks
In large-scale federated and decentralized learning, communication efficiency is one of the most challenging bottlenecks. While gossip communication -- where agents can exchange information with their connected neighbors -- is more cost-effective than communicating with the remote server, it often requires a greater number of communication rounds, especially for large and sparse networks. To tackle the trade-off, we examine the communication efficiency under a semi-decentralized communication protocol, in which agents can perform both agent-to-agent and agent-to-server communication in a probabilistic manner. We design a tailored communication-efficient algorithm over semi-decentralized networks, referred to as PISCO, which inherits the robustness to data heterogeneity thanks to gradient tracking and allows multiple local updates for saving communication. We establish the convergence rate of PISCO for nonconvex problems and show that PISCO enjoys a linear speedup in terms of the number of agents and local updates. Our numerical results highlight the superior communication efficiency of PISCO and its resilience to data heterogeneity and various network topologies.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
A Multi-Token Coordinate Descent Method for Semi-Decentralized Vertical Federated Learning
Communication efficiency is a major challenge in federated learning (FL). In client-server schemes, the server constitutes a bottleneck, and while decentralized setups spread communications, they do not necessarily reduc…
Federated LearningVertical Federated LearningFedSR: A Semi-Decentralized Federated Learning Algorithm for Non-IIDness in IoT System
In the Industrial Internet of Things (IoT), a large amount of data will be generated every day. Due to privacy and security issues, it is difficult to collect all these data together to train deep learning models, thus t…
Federated LearningWhen Decentralized Optimization Meets Federated Learning
Federated learning is a new learning paradigm for extracting knowledge from distributed data. Due to its favorable properties in preserving privacy and saving communication costs, it has been extensively studied and wide…
Federated LearningSemi-decentralized Federated Time Series Prediction with Client Availability Budgets
Federated learning (FL) effectively promotes collaborative training among distributed clients with privacy considerations in the Internet of Things (IoT) scenarios. Despite of data heterogeneity, FL clients may also be c…
Time Series PredictionFederated LearningTRAIL: Trust-Aware Client Scheduling for Semi-Decentralized Federated Learning
Due to the sensitivity of data, Federated Learning (FL) is employed to enable distributed machine learning while safeguarding data privacy and accommodating the requirements of various devices. However, in the context of…
Federated LearningScheduling