Enhancing Clinical BERT Embedding using a Biomedical Knowledge Base
Domain knowledge is important for building Natural Language Processing (NLP) systems for low-resource settings, such as in the clinical domain. In this paper, a novel joint training method is introduced for adding knowledge base information from the Unified Medical Language System (UMLS) into language model pre-training for some clinical domain corpus. We show that in three different downstream clinical NLP tasks, our pre-trained language model outperforms the corresponding model with no knowledge base information and other state-of-the-art models. Specifically, in a natural language inference task applied to clinical texts, our knowledge base pre-training approach improves accuracy by up to 1.7{\%}, whereas in clinical name entity recognition tasks, the F1-score improves by up to 1.0{\%}. The pre-trained models are available at https://github.com/noc-lab/clinical-kb-bert.
Code (1)
Tasks
Language ModelingLanguage ModellingNatural Language InferenceSimilar Papers 제목 키워드 기반
UmlsBERT: Clinical Domain Knowledge Augmentation of Contextual Embeddings Using the Unified Medical Language System Metathesaurus
Contextual word embedding models, such as BioBERT and Bio_ClinicalBERT, have achieved state-of-the-art results in biomedical natural language processing tasks by focusing their pre-training process on domain-specific cor…
named-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)Natural Language Inference+2Hierarchical Pretraining for Biomedical Term Embeddings
Electronic health records (EHR) contain narrative notes that provide extensive details on the medical condition and management of patients. Natural language processing (NLP) of clinical notes can use observed frequencies…
Decision MakingKnowledge GraphsManagementRepresentation Learning+1BERT-based Ranking for Biomedical Entity Normalization
Developing high-performance entity normalization algorithms that can alleviate the term variation problem is of great interest to the biomedical community. Although deep learning-based methods have been successfully appl…
Word EmbeddingsBioBERTpt - A Portuguese Neural Language Model for Clinical Named Entity Recognition
With the growing number of electronic health record data, clinical NLP tasks have become increasingly relevant to unlock valuable information from unstructured clinical text. Although the performance of downstream NLP ta…
Language ModelingLanguage Modellingnamed-entity-recognitionNamed Entity Recognition+3MedicalBERT: enhancing biomedical natural language processing using pretrained BERT-based model
Recent advances in natural language processing (NLP) have been driven bypretrained language models like BERT, RoBERTa, T5, and GPT. Thesemodels excel at understanding complex texts, but biomedical literature, withits dom…
Relation ExtractionQuestion AnsweringTransfer Learning