Improving Biomedical Pretrained Language Models with Knowledge
Pretrained language models have shown success in many natural language processing tasks. Many works explore incorporating knowledge into language models. In the biomedical domain, experts have taken decades of effort on building large-scale knowledge bases. For example, the Unified Medical Language System (UMLS) contains millions of entities with their synonyms and defines hundreds of relations among entities. Leveraging this knowledge can benefit a variety of downstream tasks such as named entity recognition and relation extraction. To this end, we propose KeBioLM, a biomedical pretrained language model that explicitly leverages knowledge from the UMLS knowledge bases. Specifically, we extract entities from PubMed abstracts and link them to UMLS. We then train a knowledge-aware language model that firstly applies a text-only encoding layer to learn entity representation and applies a text-entity fusion encoding to aggregate entity representation. Besides, we add two training objectives as entity detection and entity linking. Experiments on the named entity recognition and relation extraction from the BLURB benchmark demonstrate the effectiveness of our approach. Further analysis on a collected probing dataset shows that our model has better ability to model medical knowledge.
Code (1)
Tasks
Entity LinkingLanguage ModelingLanguage Modellingnamed-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)RelationRelation ExtractionSimilar Papers 제목 키워드 기반
KBioXLM: A Knowledge-anchored Biomedical Multilingual Pretrained Language Model
Most biomedical pretrained language models are monolingual and cannot handle the growing cross-lingual requirements. The scarcity of non-English domain corpora, not to mention parallel data, poses a significant hurdle in…
Language ModelingLanguage ModellingRelationRelation Prediction+1Extracting Biomedical Factual Knowledge Using Pretrained Language Model and Electronic Health Record Context
Language Models (LMs) have performed well on biomedical natural language processing applications. In this study, we conducted some experiments to use prompt methods to extract knowledge from LMs as new knowledge Bases (L…
Language ModelingLanguage ModellingBALI: Enhancing Biomedical Language Representations through Knowledge Graph and Language Model Alignment
In recent years, there has been substantial progress in using pretrained Language Models (LMs) on a range of tasks aimed at improving the understanding of biomedical texts. Nonetheless, existing biomedical LLMs show limi…
Knowledge GraphsEnriching Biomedical Knowledge for Low-resource Language Through Large-Scale Translation
Biomedical data and benchmarks are highly valuable yet very limited in low-resource languages other than English such as Vietnamese. In this paper, we make use of a state-of-the-art translation model in English-Vietnames…
DecoderTranslationAutomatic Biomedical Term Clustering by Learning Fine-grained Term Representations
Term clustering is important in biomedical knowledge graph construction. Using similarities between terms embedding is helpful for term clustering. State-of-the-art term embeddings leverage pretrained language models to …
ClusteringContrastive Learninggraph constructionKnowledge Graphs