Refined Analysis of Federated Averaging's Bias and Federated Richardson-Romberg Extrapolation
In this paper, we present a novel analysis of FedAvg with constant step size, relying on the Markov property of the underlying process. We demonstrate that the global iterates of the algorithm converge to a stationary distribution and analyze its resulting bias and variance relative to the problem's solution. We provide a first-order expansion of the bias in both homogeneous and heterogeneous settings. Interestingly, this bias decomposes into two distinct components: one that depends solely on stochastic gradient noise and another on client heterogeneity. Finally, we introduce a new algorithm based on the Richardson-Romberg extrapolation technique to mitigate this bias.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
On Principled Local Optimization Methods for Federated Learning
Federated Learning (FL), a distributed learning paradigm that scales on-device learning collaboratively, has emerged as a promising approach for decentralized AI applications. Local optimization methods such as Federated…
Federated LearningFaster On-Device Training Using New Federated Momentum Algorithm
Mobile crowdsensing has gained significant attention in recent years and has become a critical paradigm for emerging Internet of Things applications. The sensing devices continuously generate a significant quantity of da…
Federated LearningFederated Causal Inference in Healthcare: Methods, Challenges, and Applications
Federated causal inference enables multi-site treatment effect estimation without sharing individual-level data, offering a privacy-preserving solution for real-world evidence generation. However, data heterogeneity acro…
Causal InferencePrivacy PreservingFedSkip: Combatting Statistical Heterogeneity with Federated Skip Aggregation
The statistical heterogeneity of the non-independent and identically distributed (non-IID) data in local clients significantly limits the performance of federated learning. Previous attempts like FedProx, SCAFFOLD, MOON,…
Federated LearningFedCM: Federated Learning with Client-level Momentum
Federated Learning is a distributed machine learning approach which enables model training without data sharing. In this paper, we propose a new federated learning algorithm, Federated Averaging with Client-level Momentu…
Federated Learning