QuPeD: Quantized Personalization via Distillation with Applications to Federated Learning

2021-07-29 · NeurIPS 2021 12 · Kaan Ozkara, Navjot Singh, Deepesh Data, Suhas Diggavi

Traditionally, federated learning (FL) aims to train a single global model while collaboratively using multiple clients and a server. Two natural challenges that FL algorithms face are heterogeneity in data across clients and collaboration of clients with {\em diverse resources}. In this work, we introduce a \textit{quantized} and \textit{personalized} FL algorithm QuPeD that facilitates collective (personalized model compression) training via \textit{knowledge distillation} (KD) among clients who have access to heterogeneous data and resources. For personalization, we allow clients to learn \textit{compressed personalized models} with different quantization parameters and model dimensions/structures. Towards this, first we propose an algorithm for learning quantized models through a relaxed optimization problem, where quantization values are also optimized over. When each client participating in the (federated) learning process has different requirements for the compressed model (both in model dimension and precision), we formulate a compressed personalization framework by introducing knowledge distillation loss for local client objectives collaborating through a global model. We develop an alternating proximal gradient update for solving this compressed personalization problem, and analyze its convergence properties. Numerically, we validate that QuPeD outperforms competing personalized FL methods, FedAvg, and local training of clients in various heterogeneous settings.

📄 PDF Abstract BibTeX arXiv:2107.13892

Code (0)

등록된 구현이 없습니다.

Tasks

Federated Learning Knowledge Distillation Model Compression Quantization

Methods 이 논문이 사용한 방법론

Knowledge Distillation A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions.…

QuPeL: Quantized Personalization with Applications to Federated Learning

2021-02-23 · Kaan Ozkara, Navjot Singh, Deepesh Data, Suhas Diggavi

Diversity Federated Learning Quantization

FedD2S: Personalized Data-Free Federated Knowledge Distillation

2024-02-16 · Kawa Atapour, S. Jamal Seyedmohammadi, Jamshid Abouei, Arash Mohammadi 외

This paper addresses the challenge of mitigating data heterogeneity among clients within a Federated Learning (FL) framework. The model-drift issue, arising from the noniid nature of client data, often results in subopti…

Data-free Knowledge Distillation Fairness Federated Learning Knowledge Distillation+1

Personalized Federated Learning via Backbone Self-Distillation

2024-09-24 · Pengju Wang, Bochao Liu, Dan Zeng, Chenggang Yan 외

In practical scenarios, federated learning frequently necessitates training personalized models for each client using heterogeneous data. This paper proposes a backbone self-distillation approach to facilitate personaliz…

Federated Learning Personalized Federated Learning Transfer Learning

CD$^2$-pFed: Cyclic Distillation-guided Channel Decoupling for Model Personalization in Federated Learning

2022-04-08 · Yiqing Shen, Yuyin Zhou, Lequan Yu

Federated learning (FL) is a distributed learning paradigm that enables multiple clients to collaboratively learn a shared global model. Despite the recent progress, it remains challenging to deal with heterogeneous data…

Federated Learning Medical Image Analysis

CD2-pFed: Cyclic Distillation-Guided Channel Decoupling for Model Personalization in Federated Learning

2022-01-01 · CVPR 2022 1 · Yiqing Shen, Yuyin Zhou, Lequan Yu