Improved Generalization Bounds for Communication Efficient Federated Learning
This paper focuses on reducing the communication cost of federated learning by exploring generalization bounds and representation learning. We first characterize a tighter generalization bound for one-round federated learning based on local clients' generalizations and heterogeneity of data distribution (non-iid scenario). We also characterize a generalization bound in R-round federated learning and its relation to the number of local updates (local stochastic gradient descents (SGDs)). Then, based on our generalization bound analysis and our representation learning interpretation of this analysis, we show for the first time that less frequent aggregations, hence more local updates, for the representation extractor (usually corresponds to initial layers) leads to the creation of more generalizable models, particularly for non-iid scenarios. We design a novel Federated Learning with Adaptive Local Steps (FedALS) algorithm based on our generalization bound and representation learning analysis. FedALS employs varying aggregation frequencies for different parts of the model, so reduces the communication cost. The paper is followed with experimental results showing the effectiveness of FedALS.
Code (0)
등록된 구현이 없습니다.
Tasks
Federated LearningGeneralization BoundsRepresentation LearningSimilar Papers 제목 키워드 기반
Improved Information Theoretic Generalization Bounds for Distributed and Federated Learning
We consider information-theoretic bounds on expected generalization error for statistical learning problems in a networked setting. In this setting, there are $K$ nodes, each with its own independent dataset, and the mod…
Federated LearningGeneralization BoundsGeneralization Bounds for Noisy Iterative Algorithms Using Properties of Additive Noise Channels
Machine learning models trained by different optimization algorithms under different data distributions can exhibit distinct generalization behaviors. In this paper, we analyze the generalization of models trained by noi…
Federated LearningGeneralization BoundsLearning TheoryAggregation Weighting of Federated Learning via Generalization Bound Estimation
Federated Learning (FL) typically aggregates client model parameters using a weighting approach determined by sample proportions. However, this naive weighting method may lead to unfairness and degradation in model perfo…
Federated LearningGeneralization BoundsLessons from Generalization Error Analysis of Federated Learning: You May Communicate Less Often!
We investigate the generalization error of statistical learning models in a Federated Learning (FL) setting. Specifically, we study the evolution of the generalization error with the number of communication rounds $R$ be…
Federated LearningGeneralization in Federated Learning: A Conditional Mutual Information Framework
Federated learning (FL) is a widely adopted privacy-preserving distributed learning framework, yet its generalization performance remains less explored compared to centralized learning. In FL, the generalization error co…
Federated LearningPrivacy Preserving