Utilizing Synthetic Data for Medical Vision-Language Pre-training: Bypassing the Need for Real Images
Medical Vision-Language Pre-training (VLP) learns representations jointly from medical images and paired radiology reports. It typically requires large-scale paired image-text datasets to achieve effective pre-training for both the image encoder and text encoder. The advent of text-guided generative models raises a compelling question: Can VLP be implemented solely with synthetic images generated from genuine radiology reports, thereby mitigating the need for extensively pairing and curating image-text datasets? In this work, we scrutinize this very question by examining the feasibility and effectiveness of employing synthetic images for medical VLP. We replace real medical images with their synthetic equivalents, generated from authentic medical reports. Utilizing three state-of-the-art VLP algorithms, we exclusively train on these synthetic samples. Our empirical evaluation across three subsequent tasks, namely image classification, semantic segmentation and object detection, reveals that the performance achieved through synthetic data is on par with or even exceeds that obtained with real images. As a pioneering contribution to this domain, we introduce a large-scale synthetic medical image dataset, paired with anonymized real radiology reports. This alleviates the need of sharing medical images, which are not easy to curate and share in practice. The code and the dataset can be found in \href{https://github.com/cheliu-computation/MedSyn-RepLearn/tree/main}{https://github.com/cheliu-computation/MedSyn-RepLearn/tree/main}.
Code (1)
Tasks
image-classificationImage Classificationobject-detectionObject DetectionSemantic SegmentationSimilar Papers 제목 키워드 기반
An Early Investigation into the Utility of Multimodal Large Language Models in Medical Imaging
Recent developments in multimodal large language models (MLLMs) have spurred significant interest in their potential applications across various medical imaging domains. On the one hand, there is a temptation to use thes…
Medical Image AnalysisMoME: Mixture of Visual Language Medical Experts for Medical Imaging Segmentation
In this study, we propose MoME, a Mixture of Visual Language Medical Experts, for Medical Image Segmentation. MoME adapts the successful Mixture of Experts (MoE) paradigm, widely used in Large Language Models (LLMs), for…
Medical Image SegmentationMulti-modal Pre-training for Medical Vision-language Understanding and Generation: An Empirical Study with A New Benchmark
With the availability of large-scale, comprehensive, and general-purpose vision-language (VL) datasets such as MSCOCO, vision-language pre-training (VLP) has become an active area of research and proven to be effective f…
Image-text RetrievalMedical Report GenerationQuestion AnsweringRetrieval+2Can Medical Vision-Language Pre-training Succeed with Purely Synthetic Data?
Medical Vision-Language Pre-training (MedVLP) has made significant progress in enabling zero-shot tasks for medical image understanding. However, training MedVLP models typically requires large-scale datasets with paired…
zero-shot-classificationZero-Shot LearningUtilizing Large Language Models to Generate Synthetic Data to Increase the Performance of BERT-Based Neural Networks
An important issue impacting healthcare is a lack of available experts. Machine learning (ML) models could resolve this by aiding in diagnosing patients. However, creating datasets large enough to train these models is e…