Improving Communication Efficiency of Federated Distillation via Accumulating Local Updates
As an emerging federated learning paradigm, federated distillation enables communication-efficient model training by transmitting only small-scale knowledge during the learning process. To further improve the communication efficiency of federated distillation, we propose a novel technique, ALU, which accumulates multiple rounds of local updates before transferring the knowledge to the central server. ALU drastically decreases the frequency of communication in federated distillation, thereby significantly reducing the communication overhead during the training process. Empirical experiments demonstrate the substantial effect of ALU in improving the communication efficiency of federated distillation.
Code (1)
Tasks
Federated LearningSimilar Papers 제목 키워드 기반
Preserving Privacy in Federated Learning with Ensemble Cross-Domain Knowledge Distillation
Federated Learning (FL) is a machine learning paradigm where local nodes collaboratively train a central model while the training data remains decentralized. Existing FL methods typically share model parameters or employ…
Federated Learningimage-classificationImage ClassificationKnowledge Distillation+3Communication-Efficient Federated Distillation
Communication constraints are one of the major challenges preventing the wide-spread adoption of Federated Learning systems. Recently, Federated Distillation (FD), a new algorithmic paradigm for Federated Learning with f…
Federated Learningimage-classificationImage ClassificationLanguage Modeling+2Soft-Label Caching and Sharpening for Communication-Efficient Federated Distillation
Federated Learning (FL) enables collaborative model training across decentralized clients, enhancing privacy by keeping data local. Yet conventional FL, relying on frequent parameter-sharing, suffers from high communicat…
Federated LearningOptimized Federated Knowledge Distillation with Distributed Neural Architecture Search
Federated Learning (FL) enables collaborative model training without centralizing data. However, real-world deployments must simultaneously address statistical heterogeneity across client data (non-IID), system heterogen…
Neural Architecture SearchKnowledge DistillationFederated LearningFedKD: Communication Efficient Federated Learning via Knowledge Distillation
Federated learning is widely used to learn intelligent models from decentralized data. In federated learning, clients need to communicate their local model updates in each iteration of model learning. However, model upda…
Federated LearningKnowledge Distillation