Writing by Memorizing: Hierarchical Retrieval-based Medical Report Generation
Medical report generation is one of the most challenging tasks in medical image analysis. Although existing approaches have achieved promising results, they either require a predefined template database in order to retrieve sentences or ignore the hierarchical nature of medical report generation. To address these issues, we propose MedWriter that incorporates a novel hierarchical retrieval mechanism to automatically extract both report and sentence-level templates for clinically accurate report generation. MedWriter first employs the Visual-Language Retrieval~(VLR) module to retrieve the most relevant reports for the given images. To guarantee the logical coherence between sentences, the Language-Language Retrieval~(LLR) module is introduced to retrieve relevant sentences based on the previous generated description. At last, a language decoder fuses image features and features from retrieved reports and sentences to generate meaningful medical reports. We verified the effectiveness of our model by automatic evaluation and human evaluation on two datasets, i.e., Open-I and MIMIC-CXR.
Code (0)
등록된 구현이 없습니다.
Tasks
DecoderMedical Image AnalysisMedical Report GenerationRetrievalSentenceSimilar Papers 제목 키워드 기반
Hybrid Retrieval-Generation Reinforced Agent for Medical Image Report Generation
Generating long and coherent reports to describe medical images poses challenges to bridging visual patterns with informative human linguistic descriptions. We propose a novel Hybrid Retrieval-Generation Reinforced Agent…
Decision MakingReinforcement LearningRetrievalSentenceOn the Automatic Generation of Medical Imaging Reports
Medical imaging is widely used in clinical practice for diagnosis and treatment. Report-writing can be error-prone for unexperienced physicians, and time- consuming and tedious for experienced physicians. To address thes…
Medical Report GenerationMulti-Task LearningRanking Narrative Query Graphs for Biomedical Document Retrieval (Technical Report)
Keyword-based searches are today's standard in digital libraries. Yet, complex retrieval scenarios like in scientific knowledge bases, need more sophisticated access paths. Although each document somewhat contributes to …
Knowledge GraphsRetrievalExploring the Capabilities of Large Language Model Encoders for Image-Text Retrieval in Chest X-rays
Multimodal learning from paired medical images and clinical text is a central challenge in medical data-driven informatics, where effective cross-modal alignment is critical for scalable analysis and retrieval. In chest …
Contrastive LearningText Retrieval"Nothing Abnormal": Disambiguating Medical Reports via Contrastive Knowledge Infusion
Sharing medical reports is essential for patient-centered care. A recent line of work has focused on automatically generating reports with NLP methods. However, different audiences have different purposes when writing/re…