CDKT-FL: Cross-Device Knowledge Transfer using Proxy Dataset in Federated Learning
In a practical setting towards better generalization abilities of client models for realizing robust personalized Federated Learning (FL) systems, efficient model aggregation methods have been considered as a critical research objective. It is a challenging issue due to the consequences of non-i.i.d. properties of client's data, often referred to as statistical heterogeneity and small local data samples from the various data distributions. Therefore, to develop robust generalized global and personalized models, conventional FL methods need redesigning the knowledge aggregation from biased local models while considering huge divergence of learning parameters due to skewed client data. In this work, we demonstrate that the knowledge transfer mechanism is a de facto technique to achieve these objectives and develop a novel knowledge distillation-based approach to study the extent of knowledge transfer between the global model and local models. Henceforth, our method considers the suitability of transferring the outcome distribution and (or) the embedding vector of representation from trained models during cross-device knowledge transfer using a small proxy dataset in heterogeneous FL. In doing so, we alternatively perform cross-device knowledge transfer following general formulations as 1) global knowledge transfer and 2) on-device knowledge transfer. Through simulations on four federated datasets, we show the proposed method achieves significant speedups and high personalized performance of local models. Furthermore, the proposed approach offers a more stable algorithm than FedAvg during the training, with minimal communication data load when exchanging the trained model's outcomes and representation.
Code (0)
등록된 구현이 없습니다.
Tasks
Federated LearningKnowledge DistillationPersonalized Federated LearningTransfer LearningSimilar Papers 제목 키워드 기반
FedeKD: Energy-Based Gating for Robust Federated Knowledge Distillation under Heterogeneous Settings
Federated learning (FL) operates in heterogeneous environments, where variations in data distributions and asymmetric model design often result in negative transfer. While federated knowledge distillation (FKD) avoids di…
Knowledge DistillationFederated LearningTransfer Learning in Latent Contextual Bandits with Covariate Shift Through Causal Transportability
Transferring knowledge from one environment to another is an essential ability of intelligent systems. Nevertheless, when two environments are different, naively transferring all knowledge may deteriorate the performance…
Causal InferenceMulti-Armed BanditsTransfer LearningKnowledge Distillation of Black-Box Large Language Models
Given the exceptional performance of proprietary large language models (LLMs) like GPT-4, recent research has increasingly focused on boosting the capabilities of smaller models through knowledge distillation (KD) from t…
Knowledge DistillationTransfer LearningProxy-informed Bayesian transfer learning with unknown sources
Generalization outside the scope of one's training data requires leveraging prior knowledge about the effects that transfer, and the effects that don't, between different data sources. Bayesian transfer learning is a pri…
Transfer LearningFedPromo: Federated Lightweight Proxy Models at the Edge Bring New Domains to Foundation Models
Federated Learning (FL) is an established paradigm for training deep learning models on decentralized data. However, as the size of the models grows, conventional FL approaches often require significant computational res…
Knowledge DistillationImage ClassificationFederated Learning