Federated Cross-Modal Style-Aware Prompt Generation
Prompt learning has propelled vision-language models like CLIP to excel in diverse tasks, making them ideal for federated learning due to computational efficiency. However, conventional approaches that rely solely on final-layer features miss out on rich multi-scale visual cues and domain-specific style variations in decentralized client data. To bridge this gap, we introduce FedCSAP (Federated Cross-Modal Style-Aware Prompt Generation). Our framework harnesses low, mid, and high-level features from CLIP's vision encoder alongside client-specific style indicators derived from batch-level statistics. By merging intricate visual details with textual context, FedCSAP produces robust, context-aware prompt tokens that are both distinct and non-redundant, thereby boosting generalization across seen and unseen classes. Operating within a federated learning paradigm, our approach ensures data privacy through local training and global aggregation, adeptly handling non-IID class distributions and diverse domain-specific styles. Comprehensive experiments on multiple image classification datasets confirm that FedCSAP outperforms existing federated prompt learning methods in both accuracy and overall generalization.
Code (0)
등록된 구현이 없습니다.
Tasks
Computational EfficiencyImage ClassificationFederated LearningSimilar Papers 제목 키워드 기반
BadPromptFL: A Novel Backdoor Threat to Prompt-based Federated Learning in Multimodal Models
Prompt-based tuning has emerged as a lightweight alternative to full fine-tuning in large vision-language models, enabling efficient adaptation via learned contextual prompts. This paradigm has recently been extended to …
Federated LearningToward Federated Multimodal Graph Foundation Models: A Topology-Aware Multimodal Alignment Framework
Multimodal-attributed graphs (MAGs), whose nodes carry modalities such as images and text alongside topological structure, now pervade applications including social platforms, e-commerce, and biomedical networks, offerin…
Federated LearningFew-Shot LearningGraph LearningPRISM: Topology-Aware Cross-Modal Imputation for Modality-Deficient Federated Graph Learning
Multimodal federated graph learning (MM-FGL) aims to collaboratively learn from decentralized graphs with text and images. However, real-world clients may not share a common modality basis: a visual-search client may con…
Graph LearningMitigating Group-Level Fairness Disparities in Federated Visual Language Models
Visual language models (VLMs) have shown remarkable capabilities in multimodal tasks but face challenges in maintaining fairness across demographic groups, particularly when deployed in federated learning (FL) environmen…
counterfactualFairnessFederated LearningPrivacy PreservingFedTaste: Topology-Aware Structural Transfer for Multimodal Federated Learning with Missing Modalities
Multimodal Federated Learning is often challenged by arbitrary modality missingness and Non-IID data distributions, which lead to severe representation drift and hinder effective collaboration across clients. Existing me…
Federated Learning