Can Language Models be Biomedical Knowledge Bases?
Pre-trained language models (LMs) have become ubiquitous in solving various natural language processing (NLP) tasks. There has been increasing interest in what knowledge these LMs contain and how we can extract that knowledge, treating LMs as knowledge bases (KBs). While there has been much work on probing LMs in the general domain, there has been little attention to whether these powerful LMs can be used as domain-specific KBs. To this end, we create the BioLAMA benchmark, which is comprised of 49K biomedical factual knowledge triples for probing biomedical LMs. We find that biomedical LMs with recently proposed probing methods can achieve up to 18.51% Acc@5 on retrieving biomedical knowledge. Although this seems promising given the task difficulty, our detailed analyses reveal that most predictions are highly correlated with prompt templates without any subjects, hence producing similar results on each relation and hindering their capabilities to be used as domain-specific KBs. We hope that BioLAMA can serve as a challenging benchmark for biomedical factual probing.
Code (1)
Similar Papers 제목 키워드 기반
Extracting Biomedical Factual Knowledge Using Pretrained Language Model and Electronic Health Record Context
Language Models (LMs) have performed well on biomedical natural language processing applications. In this study, we conducted some experiments to use prompt methods to extract knowledge from LMs as new knowledge Bases (L…
Language ModelingLanguage ModellingImproving Biomedical Pretrained Language Models with Knowledge
Pretrained language models have shown success in many natural language processing tasks. Many works explore incorporating knowledge into language models. In the biomedical domain, experts have taken decades of effort on …
Entity LinkingLanguage ModelingLanguage Modellingnamed-entity-recognition+4Biomedical Knowledge Graph Refinement and Completion using Graph Representation Learning and Top-K Similarity Measure
Knowledge Graphs have been one of the fundamental methods for integrating heterogeneous data sources. Integrating heterogeneous data sources is crucial, especially in the biomedical domain, where central data-driven task…
Drug DiscoveryGraph Representation LearningKnowledge Graph CompletionKnowledge Graphs+2Benchmark and Best Practices for Biomedical Knowledge Graph Embeddings
Much of biomedical and healthcare data is encoded in discrete, symbolic form such as text and medical codes. There is a wealth of expert-curated biomedical domain knowledge stored in knowledge bases and ontologies, but t…
Graph EmbeddingKnowledge Graph EmbeddingKnowledge Graph EmbeddingsKnowledge Graphs+1Pre-trained language models with domain knowledge for biomedical extractive summarization
Biomedical text summarization is a critical task for comprehension of an ever-growing amount of biomedical literature. Pre-trained language models (PLMs) with transformer-based architectures have been shown to greatly im…
Extractive SummarizationPICOText Summarization