FedSWA: Improving Generalization in Federated Learning with Highly Heterogeneous Data via Momentum-Based Stochastic Controlled Weight Averaging

2025-07-26 · Liu junkang, Yuanyuan Liu, Fanhua Shang, Hongying Liu, Jin Liu, Wei Feng arxiv

For federated learning (FL) algorithms such as FedSAM, their generalization capability is crucial for real-word applications. In this paper, we revisit the generalization problem in FL and investigate the impact of data heterogeneity on FL generalization. We find that FedSAM usually performs worse than FedAvg in the case of highly heterogeneous data, and thus propose a novel and effective federated learning algorithm with Stochastic Weight Averaging (called \texttt{FedSWA}), which aims to find flatter minima in the setting of highly heterogeneous data. Moreover, we introduce a new momentum-based stochastic controlled weight averaging FL algorithm (\texttt{FedMoSWA}), which is designed to better align local and global models. Theoretically, we provide both convergence analysis and generalization bounds for \texttt{FedSWA} and \texttt{FedMoSWA}. We also prove that the optimization and generalization errors of \texttt{FedMoSWA} are smaller than those of their counterparts, including FedSAM and its variants. Empirically, experimental results on CIFAR10/100 and Tiny ImageNet demonstrate the superiority of the proposed algorithms compared to their counterparts. Open source code at: https://github.com/junkangLiu0/FedSWA.

📄 PDF Abstract BibTeX arXiv:2507.20016

Code (0)

등록된 구현이 없습니다.

Tasks

Federated Learning

Forget to Generalize: Iterative Adaptation for Generalization in Federated Learning

2026-02-04 · Abdulrahman Alotaibi, Irene Tenison, Miriam Kim, Isaac Lee 외 arxiv

The Web is naturally heterogeneous with user devices, geographic regions, browsing patterns, and contexts all leading to highly diverse, unique datasets. Federated Learning (FL) is an important paradigm for the Web becau…

Federated Learning

Personalized Federated Learning for Heterogeneous Clients with Clustered Knowledge Transfer

2021-09-16 · Yae Jee Cho, Jianyu Wang, Tarun Chiruvolu, Gauri Joshi

Personalized federated learning (FL) aims to train model(s) that can perform well for individual clients that are highly data and system heterogeneous. Most work in personalized FL, however, assumes using the same model …

Federated Learning Personalized Federated Learning Transfer Learning

Harmonizing Generalization and Personalization in Federated Prompt Learning

2024-05-16 · Tianyu Cui, Hongxia Li, Jingya Wang, Ye Shi

Federated Prompt Learning (FPL) incorporates large pre-trained Vision-Language models (VLM) into federated learning through prompt tuning. The transferable representations and remarkable generalization capacity of VLM ma…

Federated Learning Prompt Learning

Heterogeneous Ensemble Knowledge Transfer for Training Large Models in Federated Learning

2022-04-27 · Yae Jee Cho, Andre Manoel, Gauri Joshi, Robert Sim 외

Federated learning (FL) enables edge-devices to collaboratively learn a model without disclosing their private data to a central aggregating server. Most existing FL algorithms require models of identical architecture to…

Diversity Ensemble Learning Federated Learning Transfer Learning

Separate Aggregation of Split Network for Personalized Federated Learning

2026-05-26 · Yunseok Kang, Jaeyoung Song arxiv

Federated learning enables collaborative model training without sharing raw data, but its performance can degrade substantially under heterogeneous client data distributions. A single global model often cannot satisfy di…

Personalized Federated Learning