Rethinking Architecture Design for Tackling Data Heterogeneity in Federated Learning
Federated learning is an emerging research paradigm enabling collaborative training of machine learning models among different organizations while keeping data private at each institution. Despite recent progress, there remain fundamental challenges such as the lack of convergence and the potential for catastrophic forgetting across real-world heterogeneous devices. In this paper, we demonstrate that self-attention-based architectures (e.g., Transformers) are more robust to distribution shifts and hence improve federated learning over heterogeneous data. Concretely, we conduct the first rigorous empirical investigation of different neural architectures across a range of federated algorithms, real-world benchmarks, and heterogeneous data splits. Our experiments show that simply replacing convolutional networks with Transformers can greatly reduce catastrophic forgetting of previous devices, accelerate convergence, and reach a better global model, especially when dealing with heterogeneous data. We release our code and pretrained models at https://github.com/Liangqiong/ViT-FL-main to encourage future exploration in robust architectures as an alternative to current research efforts on the optimization front.
Code (1)
Tasks
Federated LearningSimilar Papers 제목 키워드 기반
Rethinking Beam Management: Generalization Limits Under Hardware Heterogeneity
Hardware heterogeneity across diverse user devices poses new challenges for beam-based communication in 5G and beyond. This heterogeneity limits the applicability of machine learning (ML)-based algorithms. This article h…
Stratify: Rethinking Federated Learning for Non-IID Data through Balanced Sampling
Federated Learning (FL) on non-independently and identically distributed (non-IID) data remains a critical challenge, as existing approaches struggle with severe data heterogeneity. Current methods primarily address symp…
Federated LearningFedShift: Tackling Dual Heterogeneity Problem of Federated Learning via Weight Shift Aggregation
Federated Learning (FL) offers a compelling method for training machine learning models with a focus on preserving data privacy. The presence of system heterogeneity and statistical heterogeneity, recognized challenges i…
DiversityFederated LearningQuantizationTackling Computational Heterogeneity in FL: A Few Theoretical Insights
The future of machine learning lies in moving data collection along with training to the edge. Federated Learning, for short FL, has been recently proposed to achieve this goal. The principle of this approach is to aggre…
Federated LearningLocal Learning Matters: Rethinking Data Heterogeneity in Federated Learning
Federated learning (FL) is a promising strategy for performing privacy-preserving, distributed learning with a network of clients (i.e., edge devices). However, the data distribution among clients is often non-IID in nat…
Federated LearningPrivacy Preserving