paper-with-me

홈 › Papers

Rephrasing Electronic Health Records for Pretraining Clinical Language Models

2024-11-28 · Jinghui Liu, Anthony Nguyen

Clinical language models are important for many applications in healthcare, but their development depends on access to extensive clinical text for pretraining. However, obtaining clinical notes from electronic health records (EHRs) at scale is challenging due to patient privacy concerns. In this study, we rephrase existing clinical notes using LLMs to generate synthetic pretraining corpora, drawing inspiration from previous work on rephrasing web data. We examine four popular small-sized LLMs (<10B) to create synthetic clinical text to pretrain both decoder-based and encoder-based language models. The method yields better results in language modeling and downstream tasks than previous synthesis approaches without referencing real clinical text. We find that augmenting original clinical notes with synthetic corpora from different LLMs improves performances even at a small token budget, showing the potential of this method to support pretraining at the institutional level or be scaled to synthesize large-scale clinical corpora.

📄 PDF Abstract BibTeX arXiv:2411.18940

Code (0)

등록된 구현이 없습니다.

Tasks

DecoderLanguage ModelingLanguage Modelling

Similar Papers 제목 키워드 기반

Learning Longitudinal Health Representations from EHR and Wearable Data

2026-01-18 · Yuanyun Zhang, Han Zhou, Li Feng, Yilin Hong 외 arxiv

Foundation models trained on electronic health records show strong performance on many clinical prediction tasks but are limited by sparse and irregular documentation. Wearable devices provide dense continuous physiologi…

MPLite: Multi-Aspect Pretraining for Mining Clinical Health Records

2024-11-17 · Eric Yang, Pengfei Hu, Xiaoxue Han, Yue Ning

The adoption of digital systems in healthcare has resulted in the accumulation of vast electronic health records (EHRs), offering valuable data for machine learning methods to predict patient health outcomes. However, si…

Prediction

Foundation Models for Clinical Records at Health System Scale

2025-07-01 · Haresh Rengaraj Rajamohan, Xiang Gao, Weicheng Zhu, Shih-Lun Huang 외 arxiv

Large-scale pretraining has transformed modeling of language and other data types, but its potential remains underexplored in healthcare with structured electronic health records (EHRs). We present a novel generative pre…

Hi-BEHRT: Hierarchical Transformer-based model for accurate prediction of clinical events using multimodal longitudinal electronic health records

2021-06-21 · Yikuan Li, Mohammad Mamouei, Gholamreza Salimi-Khorshidi, Shishir Rao 외

Electronic health records represent a holistic overview of patients' trajectories. Their increasing availability has fueled new hopes to leverage them and develop accurate risk prediction models for a wide range of disea…

Hierarchical Pretraining on Multimodal Electronic Health Records

2023-10-11 · Xiaochen Wang, Junyu Luo, Jiaqi Wang, Ziyi Yin 외

Pretraining has proven to be a powerful technique in natural language processing (NLP), exhibiting remarkable success in various NLP downstream tasks. However, in the medical domain, existing pretrained models on electro…