Improving Joint Learning of Chest X-Ray and Radiology Report by Word Region Alignment
Self-supervised learning provides an opportunity to explore unlabeled chest X-rays and their associated free-text reports accumulated in clinical routine without manual supervision. This paper proposes a Joint Image Text Representation Learning Network (JoImTeRNet) for pre-training on chest X-ray images and their radiology reports. The model was pre-trained on both the global image-sentence level and the local image region-word level for visual-textual matching. Both are bidirectionally constrained on Cross-Entropy based and ranking-based Triplet Matching Losses. The region-word matching is calculated using the attention mechanism without direct supervision about their mapping. The pre-trained multi-modal representation learning paves the way for downstream tasks concerning image and/or text encoding. We demonstrate the representation learning quality by cross-modality retrievals and multi-label classifications on two datasets: OpenI-IU and MIMIC-CXR
Code (1)
Tasks
Representation LearningSelf-Supervised LearningSentenceTripletSimilar Papers 제목 키워드 기반
Joint Modeling of Chest Radiographs and Radiology Reports for Pulmonary Edema Assessment
We propose and demonstrate a novel machine learning algorithm that assesses pulmonary edema severity from chest radiographs. While large publicly available datasets of chest radiographs and free-text radiology reports ex…
image-classificationImage ClassificationRepresentation LearningLarge Model driven Radiology Report Generation with Clinical Quality Reinforcement Learning
Radiology report generation (RRG) has attracted significant attention due to its potential to reduce the workload of radiologists. Current RRG approaches are still unsatisfactory against clinical standards. This paper in…
DecoderLanguage ModelingLanguage ModellingLarge Language Model+2Radiologist Copilot: An Agentic Framework Orchestrating Specialized Tools for Reliable Radiology Reporting
In clinical practice, radiology reporting is an essential yet complex, time-intensive, and error-prone task, particularly for 3D medical images. Existing automated approaches based on medical vision-language models prima…
Visual ReasoningMulti-Modal Multi-Agent Reinforcement Learning for Radiology Report Generation
We propose MARL-Rad, a multi-modal multi-agent reinforcement learning framework for radiology report generation that trains the entire agentic system on policy within its deployed radiology workflow. MARL-Rad addresses t…
Multi-agent Reinforcement LearningCRRG-CLIP: Automatic Generation of Chest Radiology Reports and Classification of Chest Radiographs
The complexity of stacked imaging and the massive number of radiographs make writing radiology reports complex and inefficient. Even highly experienced radiologists struggle to maintain accuracy and consistency in interp…
Classification