Unsupervised Multimodal Representation Learning across Medical Images and Reports
Joint embeddings between medical imaging modalities and associated radiology reports have the potential to offer significant benefits to the clinical community, ranging from cross-domain retrieval to conditional generation of reports to the broader goals of multimodal representation learning. In this work, we establish baseline joint embedding results measured via both local and global retrieval methods on the soon to be released MIMIC-CXR dataset consisting of both chest X-ray images and the associated radiology reports. We examine both supervised and unsupervised methods on this task and show that for document retrieval tasks with the learned representations, only a limited amount of supervision is needed to yield results comparable to those of fully-supervised methods.
Code (0)
등록된 구현이 없습니다.
Tasks
Representation LearningRetrievalSimilar Papers 제목 키워드 기반
Building RadiologyNET: Unsupervised annotation of a large-scale multimodal medical database
Background and objective: The usage of machine learning in medical diagnosis and treatment has witnessed significant growth in recent years through the development of computer-aided diagnosis systems that are often relyi…
ClusteringMedical DiagnosisSemantic SimilaritySemantic Textual SimilarityM-IDoL: Information Decomposition for Modality-Specific and Diverse Representation Learning in Medical Foundation Model
Medical foundation models (MFMs) aim to learn universal representations from multimodal medical images that can generalize effectively to diverse downstream clinical tasks. However, most existing MFMs suffer from informa…
Representation LearningUnsupervised Multimodal 3D Medical Image Registration with Multilevel Correlation Balanced Optimization
Surgical navigation based on multimodal image registration has played a significant role in providing intraoperative guidance to surgeons by showing the relative position of the target area to critical anatomical structu…
global-optimizationImage RegistrationMedical Image RegistrationUnsupervised Segmentation of 3D Medical Images Based on Clustering and Deep Representation Learning
This paper presents a novel unsupervised segmentation method for 3D medical images. Convolutional neural networks (CNNs) have brought significant advances in image segmentation. However, most of the recent methods rely o…
ClusteringImage SegmentationMedical Image SegmentationRepresentation Learning+2Multimodal Logical Inference System for Visual-Textual Entailment
A large amount of research about multimodal inference across text and vision has been recently developed to obtain visually grounded word and sentence representations. In this paper, we use logic-based representations as…
Automated Theorem ProvingNatural Language InferenceSemantic ParsingSentence