FedSkip: Combatting Statistical Heterogeneity with Federated Skip Aggregation
The statistical heterogeneity of the non-independent and identically distributed (non-IID) data in local clients significantly limits the performance of federated learning. Previous attempts like FedProx, SCAFFOLD, MOON, FedNova and FedDyn resort to an optimization perspective, which requires an auxiliary term or re-weights local updates to calibrate the learning bias or the objective inconsistency. However, in addition to previous explorations for improvement in federated averaging, our analysis shows that another critical bottleneck is the poorer optima of client models in more heterogeneous conditions. We thus introduce a data-driven approach called FedSkip to improve the client optima by periodically skipping federated averaging and scattering local models to the cross devices. We provide theoretical analysis of the possible benefit from FedSkip and conduct extensive experiments on a range of datasets to demonstrate that FedSkip achieves much higher accuracy, better aggregation efficiency and competing communication efficiency. Source code is available at: https://github.com/MediaBrain-SJTU/FedSkip.
Code (1)
Tasks
Federated LearningSimilar Papers 제목 키워드 기반
FedSkipTwin: Digital-Twin-Guided Client Skipping for Communication-Efficient Federated Learning
Communication overhead remains a primary bottleneck in federated learning (FL), particularly for applications involving mobile and IoT devices with constrained bandwidth. This work introduces FedSkipTwin, a novel client-…
Federated LearningFederated Learning with Layer Skipping: Efficient Training of Large Language Models for Healthcare NLP
Federated learning (FL) enables collaborative model training across organizations without sharing raw data, addressing crucial privacy concerns in healthcare natural language processing (NLP). However, training large lan…
Federated LearningNERPrivacy PreservingDynamicFL: Federated Learning with Dynamic Communication Resource Allocation
Federated Learning (FL) is a collaborative machine learning framework that allows multiple users to train models utilizing their local data in a distributed manner. However, considerable statistical heterogeneity in loca…
Federated LearningTowards Robust Federated Analytics via Differentially Private Measurements of Statistical Heterogeneity
Statistical heterogeneity is a measure of how skewed the samples of a dataset are. It is a common problem in the study of differential privacy that the usage of a statistically heterogeneous dataset results in a signific…
Federated Averaging Langevin Dynamics: Toward a unified theory and new algorithms
This paper focuses on Bayesian inference in a federated learning context (FL). While several distributed MCMC algorithms have been proposed, few consider the specific limitations of FL such as communication bottlenecks a…
Bayesian InferenceFederated Learning