Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization
In federated optimization, heterogeneity in the clients' local datasets and computation speeds results in large variations in the number of local updates performed by each client in each communication round. Naive weighted aggregation of such models causes objective inconsistency, that is, the global model converges to a stationary point of a mismatched objective function which can be arbitrarily different from the true objective. This paper provides a general framework to analyze the convergence of federated heterogeneous optimization algorithms. It subsumes previously proposed methods such as FedAvg and FedProx and provides the first principled understanding of the solution bias and the convergence slowdown due to objective inconsistency. Using insights from this analysis, we propose FedNova, a normalized averaging method that eliminates objective inconsistency while preserving fast error convergence.
Code (1)
Similar Papers 제목 키워드 기반
STAGE: Tackling Semantic Drift in Multimodal Federated Graph Learning
Federated graph learning (FGL) enables collaborative training on graph data across multiple clients. As graph data increasingly contain multimodal node attributes such as text and images, multimodal federated graph learn…
Graph LearningHeterogeneity-Aware Client Sampling: A Unified Solution for Consistent Federated Learning
Federated learning (FL) commonly involves clients with diverse communication and computational capabilities. Such heterogeneity can significantly distort the optimization dynamics and lead to objective inconsistency, whe…
Federated LearningClosing the Gap between Client and Global Model Performance in Heterogeneous Federated Learning
The heterogeneity of hardware and data is a well-known and studied problem in the community of Federated Learning (FL) as running under heterogeneous settings. Recently, custom-size client models trained with Knowledge D…
Federated LearningKnowledge DistillationFADE: Enabling Federated Adversarial Training on Heterogeneous Resource-Constrained Edge Devices
Federated adversarial training can effectively complement adversarial robustness into the privacy-preserving federated learning systems. However, the high demand for memory capacity and computing power makes large-scale …
Adversarial RobustnessFederated LearningPrivacy PreservingOn the Power of Adaptive Weighted Aggregation in Heterogeneous Federated Learning and Beyond
Federated averaging (FedAvg) is the most fundamental algorithm in Federated learning (FL). Previous theoretical results assert that FedAvg convergence and generalization degenerate under heterogeneous clients. However, r…
DiversityFederated Learning