ZooPFL: Exploring Black-box Foundation Models for Personalized Federated Learning
When personalized federated learning (FL) meets large foundation models, new challenges arise from various limitations in resources. In addition to typical limitations such as data, computation, and communication costs, access to the models is also often limited. This paper endeavors to solve both the challenges of limited resources and personalization. i.e., distribution shifts between clients. To do so, we propose a method named ZOOPFL that uses Zeroth-Order Optimization for Personalized Federated Learning. ZOOPFL avoids direct interference with the foundation models and instead learns to adapt its inputs through zeroth-order optimization. In addition, we employ simple yet effective linear projections to remap its predictions for personalization. To reduce the computation costs and enhance personalization, we propose input surgery to incorporate an auto-encoder with low-dimensional and client-specific embeddings. We provide theoretical support for ZOOPFL to analyze its convergence. Extensive empirical experiments on computer vision and natural language processing tasks using popular foundation models demonstrate its effectiveness for FL on black-box foundation models.
Code (1)
Tasks
Federated LearningPersonalized Federated LearningSimilar Papers 제목 키워드 기반
Personalized Recommendation Models in Federated Settings: A Survey
Federated recommender systems (FedRecSys) have emerged as a pivotal solution for privacy-aware recommendations, balancing growing demands for data security and personalized experiences. Current research efforts predomina…
Recommendation SystemsSurveyFlexible Personalized Split Federated Learning for On-Device Fine-Tuning of Foundation Models
Fine-tuning foundation models is critical for superior performance on personalized downstream tasks, compared to using pre-trained models. Collaborative learning can leverage local clients' datasets for fine-tuning, but …
Personalized Federated LearningPersonalized Federated Fine-Tuning of Vision Foundation Models for Healthcare
Foundation models open up new possibilities for the use of AI in healthcare. However, even when pre-trained on health data, they still need to be fine-tuned for specific downstream tasks. Furthermore, although foundation…
Federated LearningBi-level Personalization for Federated Foundation Models: A Task-vector Aggregation Approach
Federated foundation models represent a new paradigm to jointly fine-tune pre-trained foundation models across clients. It is still a challenge to fine-tune foundation models for a small group of new users or specialized…
Position: Federated Foundation Language Model Post-Training Should Focus on Open-Source Models
Post-training of foundation language models has emerged as a promising research domain in federated learning (FL) with the goal to enable privacy-preserving model improvements and adaptations to user's downstream tasks. …
Federated LearningLanguage ModelingLanguage ModellingPosition+1