Visual Prompt Based Personalized Federated Learning
As a popular paradigm of distributed learning, personalized federated learning (PFL) allows personalized models to improve generalization ability and robustness by utilizing knowledge from all distributed clients. Most existing PFL algorithms tackle personalization in a model-centric way, such as personalized layer partition, model regularization, and model interpolation, which all fail to take into account the data characteristics of distributed clients. In this paper, we propose a novel PFL framework for image classification tasks, dubbed pFedPT, that leverages personalized visual prompts to implicitly represent local data distribution information of clients and provides that information to the aggregation model to help with classification tasks. Specifically, in each round of pFedPT training, each client generates a local personalized prompt related to local data distribution. Then, the local model is trained on the input composed of raw data and a visual prompt to learn the distribution information contained in the prompt. During model testing, the aggregated model obtains prior knowledge of the data distributions based on the prompts, which can be seen as an adaptive fine-tuning of the aggregation model to improve model performances on different clients. Furthermore, the visual prompt can be added as an orthogonal method to implement personalization on the client for existing FL methods to boost their performance. Experiments on the CIFAR10 and CIFAR100 datasets show that pFedPT outperforms several state-of-the-art (SOTA) PFL algorithms by a large margin in various settings.
Code (0)
등록된 구현이 없습니다.
Tasks
Federated Learningimage-classificationImage ClassificationPersonalized Federated LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Prompt-based Personalized Federated Learning for Medical Visual Question Answering
We present a novel prompt-based personalized federated learning (pFL) method to address data heterogeneity and privacy concerns in traditional medical visual question answering (VQA) methods. Specifically, we regard medi…
Federated LearningMedical Visual Question AnsweringPersonalized Federated LearningQuestion Answering+2Efficient Model Personalization in Federated Learning via Client-Specific Prompt Generation
Federated learning (FL) emerges as a decentralized learning framework which trains models from multiple distributed clients without sharing their data to preserve privacy. Recently, large-scale pre-trained models (e.g., …
Federated LearningTowards Instance-wise Personalized Federated Learning via Semi-Implicit Bayesian Prompt Tuning
Federated learning (FL) is a privacy-preserving machine learning paradigm that enables collaborative model training across multiple distributed clients without disclosing their raw data. Personalized federated learning (…
Personalized Federated LearningVisual Prompt TuningFedMGP: Personalized Federated Learning with Multi-Group Text-Visual Prompts
In this paper, we introduce FedMGP, a new paradigm for personalized federated prompt learning in vision-language models. FedMGP equips each client with multiple groups of paired textual and visual prompts, enabling the m…
Personalized Federated LearningRepresentation LearningDomain GeneralizationMixture of Experts Made Personalized: Federated Prompt Learning for Vision-Language Models
Federated prompt learning benefits federated learning with CLIP-like Vision-Language Model's (VLM's) robust representation learning ability through prompt learning. However, current federated prompt learning methods are …
Federated LearningMixture-of-ExpertsPrompt LearningRepresentation Learning