Spectral Co-Distillation for Personalized Federated Learning
Personalized federated learning (PFL) has been widely investigated to address the challenge of data heterogeneity, especially when a single generic model is inadequate in satisfying the diverse performance requirements of local clients simultaneously. Existing PFL methods are inherently based on the idea that the relations between the generic global and personalized local models are captured by the similarity of model weights. Such a similarity is primarily based on either partitioning the model architecture into generic versus personalized components, or modeling client relationships via model weights. To better capture similar (yet distinct) generic versus personalized model representations, we propose \textit{spectral distillation}, a novel distillation method based on model spectrum information. Building upon spectral distillation, we also introduce a co-distillation framework that establishes a two-way bridge between generic and personalized model training. Moreover, to utilize the local idle time in conventional PFL, we propose a wait-free local training protocol. Through extensive experiments on multiple datasets over diverse heterogeneous data settings, we demonstrate the outperformance and efficacy of our proposed spectral co-distillation method, as well as our wait-free training protocol.
Code (1)
Tasks
Federated LearningPersonalized Federated LearningSimilar Papers 제목 키워드 기반
Towards Personalized Federated Learning via Comprehensive Knowledge Distillation
Federated learning is a distributed machine learning paradigm designed to protect data privacy. However, data heterogeneity across various clients results in catastrophic forgetting, where the model rapidly forgets previ…
Federated LearningKnowledge DistillationPersonalized Federated LearningPersonalized Federated Learning via Backbone Self-Distillation
In practical scenarios, federated learning frequently necessitates training personalized models for each client using heterogeneous data. This paper proposes a backbone self-distillation approach to facilitate personaliz…
Federated LearningPersonalized Federated LearningTransfer LearningPersonalized Federated Learning through Global Knowledge Distillation and Local Head Adaptation
Statistical heterogeneity limits federated learning when a single global classifier cannot represent client-specific label distributions. In this work, we propose Personalized Federated Knowledge Distillation with Head A…
Personalized Federated LearningKnowledge DistillationTailored Federated Learning: Leveraging Direction Regulation & Knowledge Distillation
Federated learning (FL) has emerged as a transformative training paradigm, particularly invaluable in privacy-sensitive domains like healthcare. However, client heterogeneity in data, computing power, and tasks poses a s…
Federated LearningKnowledge DistillationFedSSP: Federated Graph Learning with Spectral Knowledge and Personalized Preference
Personalized Federated Graph Learning (pFGL) facilitates the decentralized training of Graph Neural Networks (GNNs) without compromising privacy while accommodating personalized requirements for non-IID participants. In …
Graph Learning