Communication-Efficient Heterogeneous Federated Learning with Generalized Heavy-Ball Momentum
Federated Learning (FL) has emerged as the state-of-the-art approach for learning from decentralized data in privacy-constrained scenarios. However, system and statistical challenges hinder real-world applications, which demand efficient learning from edge devices and robustness to heterogeneity. Despite significant research efforts, existing approaches (i) are not sufficiently robust, (ii) do not perform well in large-scale scenarios, and (iii) are not communication efficient. In this work, we propose a novel Generalized Heavy-Ball Momentum (GHBM), motivating its principled application to counteract the effects of statistical heterogeneity in FL. Then, we present FedHBM as an adaptive, communication-efficient by-design instance of GHBM. Extensive experimentation on vision and language tasks, in both controlled and realistic large-scale scenarios, provides compelling evidence of substantial and consistent performance gains over the state of the art.
Code (1)
Tasks
Federated LearningSimilar Papers 제목 키워드 기반
Robust Federated Learning Over the Air: Combating Heavy-Tailed Noise with Median Anchored Clipping
Leveraging over-the-air computations for model aggregation is an effective approach to cope with the communication bottleneck in federated edge learning. By exploiting the superposition properties of multi-access channel…
Federated LearningAccelerated Convergence of Stochastic Heavy Ball Method under Anisotropic Gradient Noise
Heavy-ball momentum with decaying learning rates is widely used with SGD for optimizing deep learning models. In contrast to its empirical popularity, the understanding of its theoretical property is still quite limited,…
Federated LearningFedUHB: Accelerating Federated Unlearning via Polyak Heavy Ball Method
Federated learning facilitates collaborative machine learning, enabling multiple participants to collectively develop a shared model while preserving the privacy of individual data. The growing importance of the "right t…
Federated LearningFedGH: Heterogeneous Federated Learning with Generalized Global Header
Federated learning (FL) is an emerging machine learning paradigm that allows multiple parties to train a shared model collaboratively in a privacy-preserving manner. Existing horizontal FL methods generally assume that t…
Federated LearningPredictionPrivacy PreservingTowards Robust and Efficient Federated Low-Rank Adaptation with Heterogeneous Clients
Federated fine-tuning for Large Language Models (LLMs) faces significant challenges due to the heavy communication overhead of transmitting large model updates. Although Low Rank Adaptation (LoRA) has been proposed as a …
Federated Learning