FedDistill: Global Model Distillation for Local Model De-Biasing in Non-IID Federated Learning
Federated Learning (FL) is a novel approach that allows for collaborative machine learning while preserving data privacy by leveraging models trained on decentralized devices. However, FL faces challenges due to non-uniformly distributed (non-iid) data across clients, which impacts model performance and its generalization capabilities. To tackle the non-iid issue, recent efforts have utilized the global model as a teaching mechanism for local models. However, our pilot study shows that their effectiveness is constrained by imbalanced data distribution, which induces biases in local models and leads to a 'local forgetting' phenomenon, where the ability of models to generalize degrades over time, particularly for underrepresented classes. This paper introduces FedDistill, a framework enhancing the knowledge transfer from the global model to local models, focusing on the issue of imbalanced class distribution. Specifically, FedDistill employs group distillation, segmenting classes based on their frequency in local datasets to facilitate a focused distillation process to classes with fewer samples. Additionally, FedDistill dissects the global model into a feature extractor and a classifier. This separation empowers local models with more generalized data representation capabilities and ensures more accurate classification across all classes. FedDistill mitigates the adverse effects of data imbalance, ensuring that local models do not forget underrepresented classes but instead become more adept at recognizing and classifying them accurately. Our comprehensive experiments demonstrate FedDistill's effectiveness, surpassing existing baselines in accuracy and convergence speed across several benchmark datasets.
Code (0)
등록된 구현이 없습니다.
Tasks
Federated LearningmodelTransfer LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Optimized Federated Knowledge Distillation with Distributed Neural Architecture Search
Federated Learning (FL) enables collaborative model training without centralizing data. However, real-world deployments must simultaneously address statistical heterogeneity across client data (non-IID), system heterogen…
Neural Architecture SearchKnowledge DistillationFederated LearningPersonalized Federated Learning through Global Knowledge Distillation and Local Head Adaptation
Statistical heterogeneity limits federated learning when a single global classifier cannot represent client-specific label distributions. In this work, we propose Personalized Federated Knowledge Distillation with Head A…
Personalized Federated LearningKnowledge DistillationPreservation of the Global Knowledge by Not-True Distillation in Federated Learning
In federated learning, a strong global model is collaboratively learned by aggregating clients' locally trained models. Although this precludes the need to access clients' data directly, the global model's convergence of…
Continual LearningFederated LearningKnowledge DistillationSelf-Knowledge DistillationFederated Virtual Learning on Heterogeneous Data with Local-global Distillation
Despite Federated Learning (FL)'s trend for learning machine learning models in a distributed manner, it is susceptible to performance drops when training on heterogeneous data. In addition, FL inevitability faces the ch…
Dataset DistillationFederated LearningFedDTG:Federated Data-Free Knowledge Distillation via Three-Player Generative Adversarial Networks
While existing federated learning approaches primarily focus on aggregating local models to construct a global model, in realistic settings, some clients may be reluctant to share their private models due to the inclusio…
Data-free Knowledge DistillationFederated LearningGenerative Adversarial NetworkKnowledge Distillation