Improving Medical Report Generation with Adapter Tuning and Knowledge Enhancement in Vision-Language Foundation Models
Medical report generation demands automatic creation of coherent and precise descriptions for medical images. However, the scarcity of labelled medical image-report pairs poses formidable challenges in developing large-scale neural networks capable of harnessing the potential of artificial intelligence, exemplified by large language models. This study builds upon the state-of-the-art vision-language pre-training and fine-tuning approach, BLIP-2, to customize general large-scale foundation models. Integrating adapter tuning and a medical knowledge enhancement loss, our model significantly improves accuracy and coherence. Validation on the dataset of ImageCLEFmedical 2023 demonstrates our model's prowess, achieving the best-averaged results against several state-of-the-art methods. Significant improvements in ROUGE and CIDEr underscore our method's efficacy, highlighting promising outcomes for the rapid medical-domain adaptation of the vision-language foundation models in addressing challenges posed by data scarcity.
Code (1)
Tasks
Domain AdaptationMedical Report GenerationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
UniCrossAdapter: Multimodal Adaptation of CLIP for Radiology Report Generation
Automated radiology report generation aims to expedite the tedious and error-prone reporting process for radiologists. While recent works have made progress, learning to align medical images and textual findings remains …
Image CaptioningTransfer LearningDiversifying Knowledge Enhancement of Biomedical Language Models using Adapter Modules and Knowledge Graphs
Recent advances in natural language processing (NLP) owe their success to pre-training language models on large amounts of unstructured data. Still, there is an increasing effort to combine the unstructured nature of LMs…
Document ClassificationKnowledge GraphsNatural Language InferenceQuestion AnsweringFedPIA -- Permuting and Integrating Adapters leveraging Wasserstein Barycenters for Finetuning Foundation Models in Multi-Modal Federated Learning
Large Vision-Language Models typically require large text and image datasets for effective fine-tuning. However, collecting data from various sites, especially in healthcare, is challenging due to strict privacy regulati…
Federated Learningparameter-efficient fine-tuningQuestion AnsweringVisual Question AnsweringDiff-CXR: Report-to-CXR generation through a disease-knowledge enhanced diffusion model
Text-To-Image (TTI) generation is significant for controlled and diverse image generation with broad potential applications. Although current medical TTI methods have made some progress in report-to-Chest-Xray (CXR) gene…
Image GenerationAuto-Encoding Knowledge Graph for Unsupervised Medical Report Generation
Medical report generation, which aims to automatically generate a long and coherent report of a given medical image, has been receiving growing research interests. Existing approaches mainly adopt a supervised manner and…
DecoderMedical Report Generation