ProFe: Communication-Efficient Decentralized Federated Learning via Distillation and Prototypes
Decentralized Federated Learning (DFL) trains models in a collaborative and privacy-preserving manner while removing model centralization risks and improving communication bottlenecks. However, DFL faces challenges in efficient communication management and model aggregation within decentralized environments, especially with heterogeneous data distributions. Thus, this paper introduces ProFe, a novel communication optimization algorithm for DFL that combines knowledge distillation, prototype learning, and quantization techniques. ProFe utilizes knowledge from large local models to train smaller ones for aggregation, incorporates prototypes to better learn unseen classes, and applies quantization to reduce data transmitted during communication rounds. The performance of ProFe has been validated and compared to the literature by using benchmark datasets like MNIST, CIFAR10, and CIFAR100. Results showed that the proposed algorithm reduces communication costs by up to ~40-50% while maintaining or improving model performance. In addition, it adds ~20% training time due to increased complexity, generating a trade-off.
Code (0)
등록된 구현이 없습니다.
Tasks
Federated LearningKnowledge DistillationManagementPrivacy PreservingQuantizationSimilar Papers 제목 키워드 기반
Towards efficient compression and communication for prototype-based decentralized learning
In prototype-based federated learning, the exchange of model parameters between clients and the master server is replaced by transmission of prototypes or quantized versions of the data samples to the aggregation server.…
Data CompressionFederated LearningOn the Fast Adaptation of Delayed Clients in Decentralized Federated Learning: A Centroid-Aligned Distillation Approach
Decentralized Federated Learning (DFL) struggles with the slow adaptation of late-joining delayed clients and high communication costs in asynchronous environments. These limitations significantly hinder overall performa…
Federated LearningFederated Learning via Decentralized Dataset Distillation in Resource-Constrained Edge Environments
In federated learning, all networked clients contribute to the model training cooperatively. However, with model sizes increasing, even sharing the trained partial models often leads to severe communication bottlenecks i…
Dataset DistillationFederated LearningFedDKD: Federated Learning with Decentralized Knowledge Distillation
The performance of federated learning in neural networks is generally influenced by the heterogeneity of the data distribution. For a well-performing global model, taking a weighted average of the local models, as done b…
Federated LearningKnowledge DistillationFedKD: Communication Efficient Federated Learning via Knowledge Distillation
Federated learning is widely used to learn intelligent models from decentralized data. In federated learning, clients need to communicate their local model updates in each iteration of model learning. However, model upda…
Federated LearningKnowledge Distillation