Adaptive Distillation for Decentralized Learning from Heterogeneous Clients
This paper addresses the problem of decentralized learning to achieve a high-performance global model by asking a group of clients to share local models pre-trained with their own data resources. We are particularly interested in a specific case where both the client model architectures and data distributions are diverse, which makes it nontrivial to adopt conventional approaches such as Federated Learning and network co-distillation. To this end, we propose a new decentralized learning method called Decentralized Learning via Adaptive Distillation (DLAD). Given a collection of client models and a large number of unlabeled distillation samples, the proposed DLAD 1) aggregates the outputs of the client models while adaptively emphasizing those with higher confidence in given distillation samples and 2) trains the global model to imitate the aggregated outputs. Our extensive experimental evaluation on multiple public datasets (MNIST, CIFAR-10, and CINIC-10) demonstrates the effectiveness of the proposed method.
Code (0)
등록된 구현이 없습니다.
Tasks
Federated LearningSimilar Papers 제목 키워드 기반
Federated Incremental Semantic Segmentation
Federated learning-based semantic segmentation (FSS) has drawn widespread attention via decentralized training on local clients. However, most FSS models assume categories are fixed in advance, thus heavily undergoing fo…
Federated LearningRelationSegmentationSemantic SegmentationNo One Left Behind: Real-World Federated Class-Incremental Learning
Federated learning (FL) is a hot collaborative training framework via aggregating model parameters of decentralized local clients. However, most FL methods unreasonably assume data categories of FL framework are known an…
class-incremental learningClass Incremental LearningFederated LearningIncremental LearningDFRD: Data-Free Robustness Distillation for Heterogeneous Federated Learning
Federated Learning (FL) is a privacy-constrained decentralized machine learning paradigm in which clients enable collaborative training without compromising private data. However, how to learn a robust global model in th…
Data-free Knowledge DistillationDiversityFederated Learningimage-classification+2No One Left Behind: Inclusive Federated Learning over Heterogeneous Devices
Federated learning (FL) is an important paradigm for training global models from decentralized data in a privacy-preserving way. Existing FL methods usually assume the global model can be trained on any participating cli…
Federated LearningKnowledge DistillationPrivacy PreservingFedX: Unsupervised Federated Learning with Cross Knowledge Distillation
This paper presents FedX, an unsupervised federated learning framework. Our model learns unbiased representation from decentralized and heterogeneous local data. It employs a two-sided knowledge distillation with contras…
Contrastive LearningFederated LearningKnowledge Distillation