Selective Knowledge Sharing for Privacy-Preserving Federated Distillation without A Good Teacher
While federated learning is promising for privacy-preserving collaborative learning without revealing local data, it remains vulnerable to white-box attacks and struggles to adapt to heterogeneous clients. Federated distillation (FD), built upon knowledge distillation--an effective technique for transferring knowledge from a teacher model to student models--emerges as an alternative paradigm, which provides enhanced privacy guarantees and addresses model heterogeneity. Nevertheless, challenges arise due to variations in local data distributions and the absence of a well-trained teacher model, which leads to misleading and ambiguous knowledge sharing that significantly degrades model performance. To address these issues, this paper proposes a selective knowledge sharing mechanism for FD, termed Selective-FD. It includes client-side selectors and a server-side selector to accurately and precisely identify knowledge from local and ensemble predictions, respectively. Empirical studies, backed by theoretical insights, demonstrate that our approach enhances the generalization capabilities of the FD framework and consistently outperforms baseline methods.
Code (1)
Tasks
Federated LearningKnowledge DistillationPrivacy PreservingTransfer LearningSimilar Papers 제목 키워드 기반
Federated Continual Learning for Text Classification via Selective Inter-client Transfer
In this work, we combine the two paradigms: Federated Learning (FL) and Continual Learning (CL) for text classification task in cloud-edge continuum. The objective of Federated Continual Learning (FCL) is to improve deep…
Continual LearningFederated Learningtext-classificationText Classification+1Federated Continual Recommendation
The increasing emphasis on privacy in recommendation systems has led to the adoption of Federated Learning (FL) as a privacy-preserving solution, enabling collaborative training without sharing user data. While Federated…
Recommendation SystemsFederated LearningContinual LearningPrivacy-Preserving Federated Action Recognition via Differentially Private Selective Tuning and Efficient Communication
Federated video action recognition enables collaborative model training without sharing raw video data, yet remains vulnerable to two key challenges: \textit{model exposure} and \textit{communication overhead}. Gradients…
Action RecognitionALIGN-FL: Architecture-independent Learning through Invariant Generative component sharing in Federated Learning
We present ALIGN-FL, a novel approach to distributed learning that addresses the challenge of learning from highly disjoint data distributions through selective sharing of generative components. Instead of exchanging ful…
Federated LearningEfficient Federated Learning on Knowledge Graphs via Privacy-preserving Relation Embedding Aggregation
Federated learning (FL) can be essential in knowledge representation, reasoning, and data mining applications over multi-source knowledge graphs (KGs). A recent study FedE first proposes an FL framework that shares entit…
Entity EmbeddingsFederated LearningKnowledge GraphsLink Prediction+2