Federated Adapter on Foundation Models: An Out-Of-Distribution Approach

2025-05-02 · Yiyuan Yang, Guodong Long, Tianyi Zhou, Qinghua Lu, Shanshan Ye, Jing Jiang

As foundation models gain prominence, Federated Foundation Models (FedFM) have emerged as a privacy-preserving approach to collaboratively fine-tune models in federated learning (FL) frameworks using distributed datasets across clients. A key challenge for FedFM, given the versatile nature of foundation models, is addressing out-of-distribution (OOD) generalization, where unseen tasks or clients may exhibit distribution shifts leading to suboptimal performance. Although numerous studies have explored OOD generalization in conventional FL, these methods are inadequate for FedFM due to the challenges posed by large parameter scales and increased data heterogeneity. To address these, we propose FedOA, which employs adapter-based parameter-efficient fine-tuning methods for efficacy and introduces personalized adapters with feature distance-based regularization to align distributions and guarantee OOD generalization for each client. Theoretically, we demonstrate that the conventional aggregated global model in FedFM inherently retains OOD generalization capabilities, and our proposed method enhances the personalized model's OOD generalization through regularization informed by the global model, with proven convergence under general non-convex settings. Empirically, the effectiveness of the proposed method is validated on benchmark datasets across various NLP tasks.

📄 PDF Abstract BibTeX arXiv:2505.01075

Code (0)

등록된 구현이 없습니다.

Tasks

Federated Learning parameter-efficient fine-tuningPrivacy Preserving

Methods 이 논문이 사용한 방법론

ALIGN In the ALIGN method, visual and language representations are jointly trained from noisy image alt-text data. The image and text encoders are learned via contrastive loss…

Dual-Personalizing Adapter for Federated Foundation Models

2024-03-28 · Yiyuan Yang, Guodong Long, Tao Shen, Jing Jiang 외

Recently, foundation models, particularly large language models (LLMs), have demonstrated an impressive ability to adapt to various tasks by fine-tuning diverse instruction data. Notably, federated foundation models (Fed…

Federated Learning

Federated Adaptation for Foundation Model-based Recommendations

2024-05-08 · Chunxu Zhang, Guodong Long, Hongkuan Guo, Xiao Fang 외

With the recent success of large language models, particularly foundation models with generalization abilities, applying foundation models for recommendations becomes a new paradigm to improve existing recommendation sys…

Federated Learning model Privacy Preserving Recommendation Systems

FedPEAT: Convergence of Federated Learning, Parameter-Efficient Fine Tuning, and Emulator Assisted Tuning for Artificial Intelligence Foundation Models with Mobile Edge Computing

2023-10-26 · Terence Jie Chua, Wenhan Yu, Jun Zhao, Kwok-Yan Lam

The emergence of foundation models, including language and vision models, has reshaped AI's landscape, offering capabilities across various applications. Deploying and fine-tuning these large models, like GPT-3 and BERT,…

Deep Reinforcement Learning Edge-computing Federated Learning parameter-efficient fine-tuning

FedCLIP: Fast Generalization and Personalization for CLIP in Federated Learning

2023-02-27 · Wang Lu, Xixu Hu, Jindong Wang, Xing Xie

Federated learning (FL) has emerged as a new paradigm for privacy-preserving computation in recent years. Unfortunately, FL faces two critical challenges that hinder its actual performance: data distribution heterogeneit…

Federated Learning Privacy Preserving

Adaptive Parameterization of Deep Learning Models for Federated Learning

2023-02-06 · Morten From Elvebakken, Alexandros Iosifidis, Lukas Esterle

Federated Learning offers a way to train deep neural networks in a distributed fashion. While this addresses limitations related to distributed data, it incurs a communication overhead as the model parameters or gradient…

Deep Learning Federated Learning