Towards Model Agnostic Federated Learning Using Knowledge Distillation
Is it possible to design an universal API for federated learning using which an ad-hoc group of data-holders (agents) collaborate with each other and perform federated learning? Such an API would necessarily need to be model-agnostic i.e. make no assumption about the model architecture being used by the agents, and also cannot rely on having representative public data at hand. Knowledge distillation (KD) is the obvious tool of choice to design such protocols. However, surprisingly, we show that most natural KD-based federated learning protocols have poor performance. To investigate this, we propose a new theoretical framework, Federated Kernel ridge regression, which can capture both model heterogeneity as well as data heterogeneity. Our analysis shows that the degradation is largely due to a fundamental limitation of knowledge distillation under data heterogeneity. We further validate our framework by analyzing and designing new protocols based on KD. Their performance on real world experiments using neural networks, though still unsatisfactory, closely matches our theoretical predictions.
Code (0)
등록된 구현이 없습니다.
Tasks
Federated LearningKnowledge DistillationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Feature Distillation is the Better Choice for Model-Heterogeneous Federated Learning
Model-Heterogeneous Federated Learning (Hetero-FL) has attracted growing attention for its ability to aggregate knowledge from heterogeneous models while keeping private data locally. To better aggregate knowledge from c…
Federated LearningKnowledge DistillationPersonalized Federated Learning through Global Knowledge Distillation and Local Head Adaptation
Statistical heterogeneity limits federated learning when a single global classifier cannot represent client-specific label distributions. In this work, we propose Personalized Federated Knowledge Distillation with Head A…
Personalized Federated LearningKnowledge DistillationCOSMOS: Model-Agnostic Personalized Federated Learning with Clustered Server Models and Pseudo-Label-Only Communication
Federated learning (FL) in heterogeneous environments remains challenging because client models often differ in both architecture and data distribution. While recent approaches attempt to address this challenge through c…
Personalized Federated LearningKnowledge DistillationVertical Federated Knowledge Transfer via Representation Distillation for Healthcare Collaboration Networks
Collaboration between healthcare institutions can significantly lessen the imbalance in medical resources across various geographic areas. However, directly sharing diagnostic information between institutions is typicall…
DiagnosticFederated LearningPrivacy PreservingRepresentation Learning+1FedDTG:Federated Data-Free Knowledge Distillation via Three-Player Generative Adversarial Networks
While existing federated learning approaches primarily focus on aggregating local models to construct a global model, in realistic settings, some clients may be reluctant to share their private models due to the inclusio…
Data-free Knowledge DistillationFederated LearningGenerative Adversarial NetworkKnowledge Distillation