Characterizing Impacts of Heterogeneity in Federated Learning upon Large-Scale Smartphone Data
Federated learning (FL) is an emerging, privacy-preserving machine learning paradigm, drawing tremendous attention in both academia and industry. A unique characteristic of FL is heterogeneity, which resides in the various hardware specifications and dynamic states across the participating devices. Theoretically, heterogeneity can exert a huge influence on the FL training process, e.g., causing a device unavailable for training or unable to upload its model updates. Unfortunately, these impacts have never been systematically studied and quantified in existing FL literature. In this paper, we carry out the first empirical study to characterize the impacts of heterogeneity in FL. We collect large-scale data from 136k smartphones that can faithfully reflect heterogeneity in real-world settings. We also build a heterogeneity-aware FL platform that complies with the standard FL protocol but with heterogeneity in consideration. Based on the data and the platform, we conduct extensive experiments to compare the performance of state-of-the-art FL algorithms under heterogeneity-aware and heterogeneity-unaware settings. Results show that heterogeneity causes non-trivial performance degradation in FL, including up to 9.2% accuracy drop, 2.32x lengthened training time, and undermined fairness. Furthermore, we analyze potential impact factors and find that device failure and participant bias are two potential factors for performance degradation. Our study provides insightful implications for FL practitioners. On the one hand, our findings suggest that FL algorithm designers consider necessary heterogeneity during the evaluation. On the other hand, our findings urge system providers to design specific mechanisms to mitigate the impacts of heterogeneity.
Code (0)
등록된 구현이 없습니다.
Tasks
FairnessFederated LearningPrivacy PreservingSimilar Papers 제목 키워드 기반
HeteroSwitch: Characterizing and Taming System-Induced Data Heterogeneity in Federated Learning
Federated Learning (FL) is a practical approach to train deep learning models collaboratively across user-end devices, protecting user privacy by retaining raw data on-device. In FL, participating user-end devices are hi…
Domain GeneralizationFairnessFederated LearningTowards Fair Federated Recommendation Learning: Characterizing the Inter-Dependence of System and Data Heterogeneity
Federated learning (FL) is an effective mechanism for data privacy in recommender systems by running machine learning model training on-device. While prior FL optimizations tackled the data and system heterogeneity chall…
FairnessFederated LearningModel OptimizationRecommendation SystemsOn the Impact of Device and Behavioral Heterogeneity in Federated Learning
Federated learning (FL) is becoming a popular paradigm for collaborative learning over distributed, private datasets owned by non-trusting entities. FL has seen successful deployment in production environments, and it ha…
FairnessFederated LearningRethinking Data Heterogeneity in Federated Learning: Introducing a New Notion and Standard Benchmarks
Though successful, federated learning presents new challenges for machine learning, especially when the issue of data heterogeneity, also known as Non-IID data, arises. To cope with the statistical heterogeneity, previou…
Federated LearningFederated Averaging Langevin Dynamics: Toward a unified theory and new algorithms
This paper focuses on Bayesian inference in a federated learning context (FL). While several distributed MCMC algorithms have been proposed, few consider the specific limitations of FL such as communication bottlenecks a…
Bayesian InferenceFederated Learning