Textual Inversion and Self-supervised Refinement for Radiology Report Generation
Existing mainstream approaches follow the encoder-decoder paradigm for generating radiology reports. They focus on improving the network structure of encoders and decoders, which leads to two shortcomings: overlooking the modality gap and ignoring report content constraints. In this paper, we proposed Textual Inversion and Self-supervised Refinement (TISR) to address the above two issues. Specifically, textual inversion can project text and image into the same space by representing images as pseudo words to eliminate the cross-modeling gap. Subsequently, self-supervised refinement refines these pseudo words through contrastive loss computation between images and texts, enhancing the fidelity of generated reports to images. Notably, TISR is orthogonal to most existing methods, plug-and-play. We conduct experiments on two widely-used public datasets and achieve significant improvements on various baselines, which demonstrates the effectiveness and generalization of TISR. The code will be available soon.
Code (0)
등록된 구현이 없습니다.
Tasks
DecoderMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
SERPENT-VLM : Self-Refining Radiology Report Generation Using Vision Language Models
Radiology Report Generation (R2Gen) demonstrates how Multi-modal Large Language Models (MLLMs) can automate the creation of accurate and coherent radiological reports. Existing methods often hallucinate details in text-b…
Causal Language ModelingHallucinationLanguage ModelingLanguage ModellingSelf-Supervised Contextual Language Representation of Radiology Reports to Improve the Identification of Communication Urgency
Machine learning methods have recently achieved high-performance in biomedical text analysis. However, a major bottleneck in the widespread application of these methods is obtaining the required large amounts of annotate…
Self-Supervised LearningStructure is Supervision: Multiview Masked Autoencoders for Radiology
Building robust medical machine learning systems requires pretraining strategies that exploit the intrinsic structure present in clinical data. We introduce Multiview Masked Autoencoder (MVMAE), a self-supervised framewo…
Image ReconstructionMaking the Most of Text Semantics to Improve Biomedical Vision--Language Processing
Multi-modal data abounds in biomedicine, such as radiology images and reports. Interpreting this data at scale is essential for improving clinical care and accelerating clinical research. Biomedical text with its complex…
Contrastive LearningLanguage ModelingLanguage ModellingMedical Image Classification+3Image Diffusion Models Exhibit Emergent Temporal Propagation in Videos
Image diffusion models, though originally developed for image generation, implicitly capture rich semantic structures that enable various recognition and localization tasks beyond synthesis. In this work, we investigate …
Video Object SegmentationImage GenerationObject Tracking