Graph-Text Multi-Modal Pre-training for Medical Representation Learning
As the volume of Electronic Health Records (EHR) sharply grows, there has been emerging interest in learning the representation of EHR for healthcare applications. Representation learning of EHR requires appropriate modeling of the two dominant modalities in EHR: structured data and unstructured text. In this paper, we present MedGTX, a pre-trained model for multi-modal representation learning of the structured and textual EHR data. MedGTX uses a novel graph encoder to exploit the graphical nature of structured EHR data, and a text encoder to handle unstructured text, and a cross-modal encoder to learn a joint representation space. We pre-train our model through four proxy tasks on MIMIC-III, an open-source EHR data, and evaluate our model on two clinical benchmarks and three novel downstream tasks which tackle real-world problems in EHR data. The results consistently show the effectiveness of pre-training the model for joint representation of both structured and unstructured information from EHR. Given the promising performance of MedGTX, we believe this work opens a new door to jointly understanding the two fundamental modalities of EHR data.
Code (1)
Tasks
Representation LearningSimilar Papers 제목 키워드 기반
PMC-InterCPT: Rethinking Biomedical Interleaved Data for Multimodal Continued Pretraining
Large-scale biomedical image-text datasets extracted from scientific literature provide valuable resources for medical multimodal model training. These datasets are commonly organized as image-caption pairs; however, fig…
CMKL: Modality-Aware Continual Learning for Evolving Biomedical Knowledge Graphs
Biomedical knowledge graphs are increasingly large, dynamic, and multimodal, driven by rapid advances in biotechnology such as high-throughput sequencing. Machine learning models can infer previously unobserved biomedica…
Knowledge Graph EmbeddingContinual LearningKnowledge GraphsLoGra-Med: Long Context Multi-Graph Alignment for Medical Vision-Language Model
State-of-the-art medical multi-modal large language models (med-MLLM), like LLaVA-Med or BioMedGPT, leverage instruction-following data in pre-training. However, those models primarily focus on scaling the model size and…
image-classificationImage ClassificationInstruction FollowingLanguage Modeling+4PyTDC: A multimodal machine learning training, evaluation, and inference platform for biomedical foundation models
Existing biomedical benchmarks do not provide end-to-end infrastructure for training, evaluation, and inference of models that integrate multimodal biological data and a broad range of machine learning tasks in therapeut…
BenchmarkingGraph Representation LearningRepresentation LearningMultimodal Graph-based Transformer Framework for Biomedical Relation Extraction
The recent advancement of pre-trained Transformer models has propelled the development of effective text mining models across various biomedical tasks. However, these models are primarily learned on the textual data and …
RelationRelation ExtractionSentence