Disease-informed Adaptation of Vision-Language Models
In medical image analysis, the expertise scarcity and the high cost of data annotation limits the development of large artificial intelligence models. This paper investigates the potential of transfer learning with pre-trained vision-language models (VLMs) in this domain. Currently, VLMs still struggle to transfer to the underrepresented diseases with minimal presence and new diseases entirely absent from the pretraining dataset. We argue that effective adaptation of VLMs hinges on the nuanced representation learning of disease concepts. By capitalizing on the joint visual-linguistic capabilities of VLMs, we introduce disease-informed contextual prompting in a novel disease prototype learning framework. This approach enables VLMs to grasp the concepts of new disease effectively and efficiently, even with limited data. Extensive experiments across multiple image modalities showcase notable enhancements in performance compared to existing techniques.
Code (1)
Tasks
Medical Image AnalysisRepresentation LearningTransfer LearningSimilar Papers 제목 키워드 기반
LI-TTA: Language Informed Test-Time Adaptation for Automatic Speech Recognition
Test-Time Adaptation (TTA) has emerged as a crucial solution to the domain shift challenge, wherein the target environment diverges from the original training environment. A prime exemplification is TTA for Automatic Spe…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Language ModelingLanguage Modelling+3Multi-View Synergistic Learning with Vision-Language Adaption for Low-Resource Biomedical Image Classification
Accurate biomedical image classification under low-resource conditions remains challenging due to limited annotations, subtle inter-class visual differences, and complex disease semantics. While vision--language models o…
parameter-efficient fine-tuningRepresentation LearningContrastive LearningImage ClassificationBiomedAP: A Vision-Informed Dual-Anchor Framework with Gated Cross-Modal Fusion for Robust Medical Vision-Language Adaptation
Biomedical Vision--Language Models (VLMs) have shown remarkable promise in few-shot medical diagnosis but face a critical bottleneck: \textit{fragility to prompt variations}.Existing adaptation frameworks typically optim…
parameter-efficient fine-tuningFew-Shot LearningMedical DiagnosisAn integrated language-vision foundation model for conversational diagnostics and triaging in primary eye care
Current deep learning models are mostly task specific and lack a user-friendly interface to operate. We present Meta-EyeFM, a multi-function foundation model that integrates a large language model (LLM) with vision found…
DiagnosticLanguage ModelingLanguage ModellingLarge Language ModelSemi-Supervised Few-Shot Adaptation of Vision-Language Models
Vision-language models (VLMs) pre-trained on large, heterogeneous data sources are becoming increasingly popular, providing rich multi-modal embeddings that enable efficient transfer to new tasks. A particularly relevant…
Few-Shot Image Classification