Multi-Modal One-Shot Federated Ensemble Learning for Medical Data with Vision Large Language Model
Federated learning (FL) has attracted considerable interest in the medical domain due to its capacity to facilitate collaborative model training while maintaining data privacy. However, conventional FL methods typically necessitate multiple communication rounds, leading to significant communication overhead and delays, especially in environments with limited bandwidth. One-shot federated learning addresses these issues by conducting model training and aggregation in a single communication round, thereby reducing communication costs while preserving privacy. Among these, one-shot federated ensemble learning combines independently trained client models using ensemble techniques such as voting, further boosting performance in non-IID data scenarios. On the other hand, existing machine learning methods in healthcare predominantly use unimodal data (e.g., medical images or textual reports), which restricts their diagnostic accuracy and comprehensiveness. Therefore, the integration of multi-modal data is proposed to address these shortcomings. In this paper, we introduce FedMME, an innovative one-shot multi-modal federated ensemble learning framework that utilizes multi-modal data for medical image analysis. Specifically, FedMME capitalizes on vision large language models to produce textual reports from medical images, employs a BERT model to extract textual features from these reports, and amalgamates these features with visual features to improve diagnostic accuracy. Experimental results show that our method demonstrated superior performance compared to existing one-shot federated learning methods in healthcare scenarios across four datasets with various data distributions. For instance, it surpasses existing one-shot federated learning approaches by more than 17.5% in accuracy on the RSNA dataset when applying a Dirichlet distribution with ($\alpha$ = 0.3).
Code (0)
등록된 구현이 없습니다.
Tasks
DiagnosticEnsemble LearningFederated LearningLanguage ModelingLanguage ModellingLarge Language ModelMedical Image AnalysisMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
FedBEns: One-Shot Federated Learning based on Bayesian Ensemble
One-Shot Federated Learning (FL) is a recent paradigm that enables multiple clients to cooperatively learn a global model in a single round of communication with a central server. In this paper, we analyze the One-Shot F…
Bayesian InferenceFederated LearningFedBiCross: Personalized One-Shot Federated Learning on Medical Images
Data-free knowledge distillation-based one-shot federated learning (OSFL) trains a model in a single communication round without sharing raw data, making OSFL attractive for privacy-sensitive medical applications. Howeve…
Data-free Knowledge DistillationFederated LearningData-Free Diversity-Based Ensemble Selection For One-Shot Federated Learning in Machine Learning Model Market
The emerging availability of trained machine learning models has put forward the novel concept of Machine Learning Model Market in which one can harness the collective intelligence of multiple well-trained models to impr…
DiversityEnsemble LearningFederated LearningPartially Supervised Unpaired Multi-Modal Learning for Label-Efficient Medical Image Segmentation
Unpaired Multi-Modal Learning (UMML) which leverages unpaired multi-modal data to boost model performance on each individual modality has attracted a lot of research interests in medical image analysis. However, existing…
Image SegmentationMedical Image AnalysisMedical Image SegmentationOrgan Segmentation+3Federated Self-Supervised Learning for One-Shot Cross-Modal and Cross-Imaging Technique Segmentation
Decentralized federated learning enables learning of data representations from multiple sources without compromising the privacy of the clients. In applications like medical image segmentation, where obtaining a large an…
Federated LearningImage SegmentationMedical Image SegmentationOne-Shot Segmentation+3