Ensemble Attention Distillation for Privacy-Preserving Federated Learning
We consider the problem of Federated Learning (FL) where numerous decentralized computational nodes collaborate with each other to train a centralized machine learning model without explicitly sharing their local data samples. Such decentralized training naturally leads to issues of imbalanced or differing data distributions among the local models and challenges in fusing them into a central model. Existing FL methods deal with these issues by either sharing local parameters or fusing models via online distillation. However, such a design leads to multiple rounds of inter-node communication resulting in substantial bandwidth consumption, while also increasing the risk of data leakage and consequent privacy issues. To address these problems, we propose a new distillation-based FL framework that can preserve privacy by design, while also consuming substantially less network communication resources when compared to the current state-of-the-art. Our framework engages in inter-node communication using only publicly available and approved datasets, thereby giving explicit privacy control to the user. To distill knowledge among the various local models, our framework involves a novel ensemble distillation algorithm that uses both final prediction as well as model attention. This algorithm explicitly considers the diversity among various local nodes while also seeking consensus among them. This results in a comprehensive technique to distill knowledge from various decentralized nodes. We demonstrate the various aspects and the associated benefits of our FL framework through extensive experiments that produce state-of-the-art results on both classification and segmentation tasks on natural and medical images.
Code (0)
등록된 구현이 없습니다.
Tasks
Federated LearningPrivacy PreservingSimilar Papers 제목 키워드 기반
Federated Learning with Privacy-Preserving Ensemble Attention Distillation
Federated Learning (FL) is a machine learning paradigm where many local nodes collaboratively train a central model while keeping the training data decentralized. This is particularly relevant for clinical applications s…
Federated Learningimage-classificationImage ClassificationKnowledge Distillation+1Preserving Privacy in Federated Learning with Ensemble Cross-Domain Knowledge Distillation
Federated Learning (FL) is a machine learning paradigm where local nodes collaboratively train a central model while the training data remains decentralized. Existing FL methods typically share model parameters or employ…
Federated Learningimage-classificationImage ClassificationKnowledge Distillation+3FedED: Federated Learning via Ensemble Distillation for Medical Relation Extraction
Unlike other domains, medical texts are inevitably accompanied by private information, so sharing or copying these texts is strictly restricted. However, training a medical relation extraction model requires collecting t…
Federated LearningKnowledge DistillationMedical Relation ExtractionPrivacy Preserving+2Selective Knowledge Sharing for Privacy-Preserving Federated Distillation without A Good Teacher
While federated learning is promising for privacy-preserving collaborative learning without revealing local data, it remains vulnerable to white-box attacks and struggles to adapt to heterogeneous clients. Federated dist…
Federated LearningKnowledge DistillationPrivacy PreservingTransfer LearningTowards Communication-Efficient and Privacy-Preserving Federated Representation Learning
This paper investigates the feasibility of federated representation learning under the constraints of communication cost and privacy protection. Existing works either conduct annotation-guided local training which requir…
Contrastive LearningFederated LearningPrivacy PreservingRepresentation Learning