Linear Speedup in Personalized Collaborative Learning
Collaborative training can improve the accuracy of a model for a user by trading off the model's bias (introduced by using data from other users who are potentially different) against its variance (due to the limited amount of data on any single user). In this work, we formalize the personalized collaborative learning problem as a stochastic optimization of a task 0 while giving access to N related but different tasks 1,..., N. We provide convergence guarantees for two algorithms in this setting -- a popular collaboration method known as weighted gradient averaging, and a novel bias correction method -- and explore conditions under which we can achieve linear speedup w.r.t. the number of auxiliary tasks N. Further, we also empirically study their performance confirming our theoretical insights.
Code (1)
Tasks
Federated LearningStochastic OptimizationSimilar Papers 제목 키워드 기반
Personalized Collaborative Learning with Affinity-Based Variance Reduction
Multi-agent learning faces a fundamental tension: leveraging distributed collaboration without sacrificing the personalization needed for diverse agents. This tension intensifies when aiming for full personalization whil…
Federated LearningDePRL: Achieving Linear Convergence Speedup in Personalized Decentralized Learning with Shared Representations
Decentralized learning has emerged as an alternative method to the popular parameter-server framework which suffers from high communication burden, single-point failure and scalability issues due to the need of a central…
Learning TheoryRepresentation LearningOn the Linear Speedup of Personalized Federated Reinforcement Learning with Shared Representations
Federated reinforcement learning (FedRL) enables multiple agents to collaboratively learn a policy without sharing their local trajectories collected during agent-environment interactions. However, in practice, the envir…
Sparse Federated Learning with Hierarchical Personalized Models
Federated learning (FL) can achieve privacy-safe and reliable collaborative training without collecting users' private data. Its excellent privacy security potential promotes a wide range of FL applications in Internet-o…
Autonomous VehiclesFederated LearningPersonalized Federated Learning with Moreau Envelopes
Federated learning (FL) is a decentralized and privacy-preserving machine learning technique in which a group of clients collaborate with a server to learn a global model without sharing clients' data. One challenge asso…
DiversityFederated LearningMeta-LearningModel Optimization+3