RaDialog: A Large Vision-Language Model for Radiology Report Generation and Conversational Assistance
Conversational AI tools that can generate and discuss clinically correct radiology reports for a given medical image have the potential to transform radiology. Such a human-in-the-loop radiology assistant could facilitate a collaborative diagnostic process, thus saving time and improving the quality of reports. Towards this goal, we introduce RaDialog, the first thoroughly evaluated and publicly available large vision-language model for radiology report generation and interactive dialog. RaDialog effectively integrates visual image features and structured pathology findings with a large language model (LLM) while simultaneously adapting it to a specialized domain using parameter-efficient fine-tuning. To keep the conversational abilities of the underlying LLM, we propose a comprehensive, semi-automatically labeled, image-grounded instruct dataset for chest X-ray radiology tasks. By training with this dataset, our method achieves state-of-the-art clinical correctness in report generation and shows impressive abilities in interactive tasks such as correcting reports and answering questions, serving as a foundational step toward clinical dialog systems. Our code is available on github: https://github.com/ChantalMP/RaDialog.
Code (1)
Tasks
DiagnosticLanguage ModelingLanguage ModellingLarge Language Modelparameter-efficient fine-tuningSimilar Papers 제목 키워드 기반
Semantic Consistency-Based Uncertainty Quantification for Factuality in Radiology Report Generation
Radiology report generation (RRG) has shown great potential in assisting radiologists by automating the labor-intensive task of report writing. While recent advancements have improved the quality and coherence of generat…
DiagnosticSentenceUncertainty QuantificationTRRG: Towards Truthful Radiology Report Generation With Cross-modal Disease Clue Enhanced Large Language Model
The vision-language modeling capability of multi-modal large language models has attracted wide attention from the community. However, in medical domain, radiology report generation using vision-language models still fac…
Contrastive LearningLanguage ModelingLanguage ModellingLarge Language ModelGla-AI4BioMed at RRG24: Visual Instruction-tuned Adaptation for Radiology Report Generation
We introduce a radiology-focused visual language model designed to generate radiology reports from chest X-rays. Building on previous findings that large language models (LLMs) can acquire multimodal capabilities when al…
Language ModelingLanguage ModellingUniCrossAdapter: Multimodal Adaptation of CLIP for Radiology Report Generation
Automated radiology report generation aims to expedite the tedious and error-prone reporting process for radiologists. While recent works have made progress, learning to align medical images and textual findings remains …
Image CaptioningTransfer LearningAHIVE: Anatomy-aware Hierarchical Vision Encoding for Interactive Radiology Report Retrieval
Automatic radiology report generation using deep learning models has been recently explored and found promising. Neural decoders are commonly used for the report generation where irrelevant and unfaithful contents ar…
AnatomyDiagnosticRetrieval