Optimal Model Averaging: Towards Personalized Collaborative Learning
In federated learning, differences in the data or objectives between the participating nodes motivate approaches to train a personalized machine learning model for each node. One such approach is weighted averaging between a locally trained model and the global model. In this theoretical work, we study weighted model averaging for arbitrary scalar mean estimation problems under minimal assumptions on the distributions. In a variant of the bias-variance trade-off, we find that there is always some positive amount of model averaging that reduces the expected squared error compared to the local model, provided only that the local model has a non-zero variance. Further, we quantify the (possibly negative) benefit of weighted model averaging as a function of the weight used and the optimal weight. Taken together, this work formalizes an approach to quantify the value of personalization in collaborative learning and provides a framework for future research to test the findings in multivariate parameter estimation and under a range of assumptions.
Code (0)
등록된 구현이 없습니다.
Tasks
Federated Learningmodelparameter estimationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
WAFFLE: Weighted Averaging for Personalized Federated Learning
In federated learning, model personalization can be a very effective strategy to deal with heterogeneous training data across clients. We introduce WAFFLE (Weighted Averaging For Federated LEarning), a personalized colla…
Federated LearningPersonalized Federated LearningCollaborative Learning in the Jungle (Decentralized, Byzantine, Heterogeneous, Asynchronous and Nonconvex Learning)
We study Byzantine collaborative learning, where $n$ nodes seek to collectively learn from each others' local data. The data distribution may vary from one node to another. No node is trusted, and $f < n$ nodes can behav…
Linear Speedup in Personalized Collaborative Learning
Collaborative training can improve the accuracy of a model for a user by trading off the model's bias (introduced by using data from other users who are potentially different) against its variance (due to the limited amo…
Federated LearningStochastic OptimizationAdaptive Personalized Federated Learning via Multi-task Averaging of Kernel Mean Embeddings
Personalized Federated Learning (PFL) enables a collection of agents to collaboratively learn individual models without sharing raw data. We propose a new PFL approach in which each agent optimizes a weighted combination…
Personalized Federated LearningCollaborative Chinese Text Recognition with Personalized Federated Learning
In Chinese text recognition, to compensate for the insufficient local data and improve the performance of local few-shot character recognition, it is often necessary for one organization to collect a large amount of data…
Federated LearningPersonalized Federated LearningPrivacy Preserving