Exploring Low-Resource Medical Image Classification with Weakly Supervised Prompt Learning
Most advances in medical image recognition supporting clinical auxiliary diagnosis meet challenges due to the low-resource situation in the medical field, where annotations are highly expensive and professional. This low-resource problem can be alleviated by leveraging the transferable representations of large-scale pre-trained vision-language models via relevant medical text prompts. However, existing pre-trained vision-language models require domain experts to carefully design the medical prompts, which greatly increases the burden on clinicians. To address this problem, we propose a weakly supervised prompt learning method MedPrompt to automatically generate medical prompts, which includes an unsupervised pre-trained vision-language model and a weakly supervised prompt learning model. The unsupervised pre-trained vision-language model utilizes the natural correlation between medical images and corresponding medical texts for pre-training, without any manual annotations. The weakly supervised prompt learning model only utilizes the classes of images in the dataset to guide the learning of the specific class vector in the prompt, while the learning of other context vectors in the prompt requires no manual annotations for guidance. To the best of our knowledge, this is the first model to automatically generate medical prompts. With these prompts, the pre-trained vision-language model can be freed from the strong expert dependency of manual annotation and manual prompt design. Experimental results show that the model using our automatically generated prompts outperforms its full-shot learning hand-crafted prompts counterparts with only a minimal number of labeled samples for few-shot learning, and reaches superior or comparable accuracy on zero-shot image classification. The proposed prompt generator is lightweight and therefore can be embedded into any network architecture.
Code (0)
등록된 구현이 없습니다.
Tasks
Few-Shot Learningimage-classificationImage ClassificationLanguage ModelingLanguage ModellingMedical Image ClassificationPrompt LearningZero-Shot Image ClassificationSimilar Papers 제목 키워드 기반
Exploring Weakly Supervised Semantic Segmentation Ensembles for Medical Imaging Systems
Reliable classification and detection of certain medical conditions, in images, with state-of-the-art semantic segmentation networks, require vast amounts of pixel-wise annotation. However, the public availability of suc…
SegmentationSemantic SegmentationWeakly supervised Semantic SegmentationWeakly-Supervised Semantic SegmentationWeakly-supervised Generative Adversarial Networks for medical image classification
Weakly-supervised learning has become a popular technology in recent years. In this paper, we propose a novel medical image classification algorithm, called Weakly-Supervised Generative Adversarial Networks (WSGAN), whic…
ClassificationContrastive Learningimage-classificationImage Classification+2Self-Transfer Learning for Fully Weakly Supervised Object Localization
Recent advances of deep learning have achieved remarkable performances in various challenging computer vision tasks. Especially in object localization, deep convolutional neural networks outperform traditional approaches…
Medical Image AnalysisObjectObject LocalizationTransfer Learning+2CaCL: Class-aware Codebook Learning for Weakly Supervised Segmentation on Diffuse Image Patterns
Weakly supervised learning has been rapidly advanced in biomedical image analysis to achieve pixel-wise labels (segmentation) from image-wise annotations (classification), as biomedical images naturally contain image-wis…
Image ReconstructionSegmentationWeakly-supervised LearningWeakly supervised segmentationDeep Superpixel Generation and Clustering for Weakly Supervised Segmentation of Brain Tumors in MR Images
Training machine learning models to segment tumors and other anomalies in medical images is an important step for developing diagnostic tools but generally requires manually annotated ground truth segmentations, which ne…
Binary ClassificationBrain Tumor SegmentationClusteringDiagnostic+4