Preconditioned Federated Learning
Federated Learning (FL) is a distributed machine learning approach that enables model training in communication efficient and privacy-preserving manner. The standard optimization method in FL is Federated Averaging (FedAvg), which performs multiple local SGD steps between communication rounds. FedAvg has been considered to lack algorithm adaptivity compared to modern first-order adaptive optimizations. In this paper, we propose new communication-efficient FL algortithms based on two adaptive frameworks: local adaptivity (PreFed) and server-side adaptivity (PreFedOp). Proposed methods adopt adaptivity by using a novel covariance matrix preconditioner. Theoretically, we provide convergence guarantees for our algorithms. The empirical experiments show our methods achieve state-of-the-art performances on both i.i.d. and non-i.i.d. settings.
Code (0)
등록된 구현이 없습니다.
Tasks
Federated LearningPrivacy PreservingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
FedPM: Federated Learning Using Second-order Optimization with Preconditioned Mixing of Local Parameters
We propose Federated Preconditioned Mixing (FedPM), a novel Federated Learning (FL) method that leverages second-order optimization. Prior methods--such as LocalNewton, LTDA, and FedSophia--have incorporated second-order…
Federated LearningTaming Preconditioner Drift: Unlocking the Potential of Second-Order Optimizers for Federated Learning on Non-IID Data
Second-order optimizers can significantly accelerate large-scale training, yet their naive federated variants are often unstable or even diverge on non-IID data. We show that a key culprit is \emph{preconditioner drift}:…
Federated LearningImproving Implicit Regularization of SGD with Preconditioning for Least Square Problems
Stochastic gradient descent (SGD) exhibits strong algorithmic regularization effects in practice and plays an important role in the generalization of modern machine learning. However, prior research has revealed instance…
regressionA Comparative Framework for Preconditioned Lasso Algorithms
The Lasso is a cornerstone of modern multivariate data analysis, yet its performance suffers in the common situation in which covariates are correlated. This limitation has led to a growing number of \emph{Preconditioned…
Preconditioned inexact fixed point iteration method for solving tensor absolute value equation
Preconditioning techniques are the most used methods to accelerate the tensor splitting iteration method for solving multi-linear systems. In this paper, we consider the numerical method for solving tensor absolute val…