Diversifying Knowledge Enhancement of Biomedical Language Models using Adapter Modules and Knowledge Graphs
Recent advances in natural language processing (NLP) owe their success to pre-training language models on large amounts of unstructured data. Still, there is an increasing effort to combine the unstructured nature of LMs with structured knowledge and reasoning. Particularly in the rapidly evolving field of biomedical NLP, knowledge-enhanced language models (KELMs) have emerged as promising tools to bridge the gap between large language models and domain-specific knowledge, considering the available biomedical knowledge graphs (KGs) curated by experts over the decades. In this paper, we develop an approach that uses lightweight adapter modules to inject structured biomedical knowledge into pre-trained language models (PLMs). We use two large KGs, the biomedical knowledge system UMLS and the novel biochemical ontology OntoChem, with two prominent biomedical PLMs, PubMedBERT and BioLinkBERT. The approach includes partitioning knowledge graphs into smaller subgraphs, fine-tuning adapter modules for each subgraph, and combining the knowledge in a fusion layer. We test the performance on three downstream tasks: document classification,question answering, and natural language inference. We show that our methodology leads to performance improvements in several instances while keeping requirements in computing power low. Finally, we provide a detailed interpretation of the results and report valuable insights for future work.
Code (0)
등록된 구현이 없습니다.
Tasks
Document ClassificationKnowledge GraphsNatural Language InferenceQuestion AnsweringMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Pre-trained language models with domain knowledge for biomedical extractive summarization
Biomedical text summarization is a critical task for comprehension of an ever-growing amount of biomedical literature. Pre-trained language models (PLMs) with transformer-based architectures have been shown to greatly im…
Extractive SummarizationPICOText SummarizationParameter-Efficient Domain Knowledge Integration from Multiple Sources for Biomedical Pre-trained Language Models
Domain-specific pre-trained language models (PLMs) have achieved great success over various downstream tasks in different domains. However, existing domain-specific PLMs mostly rely on self-supervised learning over large…
ArticlesSelf-Supervised LearningMedAdapter: Efficient Test-Time Adaptation of Large Language Models towards Medical Reasoning
Despite their improved capabilities in generation and reasoning, adapting large language models (LLMs) to the biomedical domain remains challenging due to their immense size and corporate privacy. In this work, we propos…
Privacy PreservingTest-time AdaptationAdapter-based Approaches to Knowledge-enhanced Language Models -- A Survey
Knowledge-enhanced language models (KELMs) have emerged as promising tools to bridge the gap between large-scale language models and domain-specific knowledge. KELMs can achieve higher factual accuracy and mitigate hallu…
General KnowledgeKnowledge GraphsSurveySystematic Literature ReviewMixture-of-Partitions: Infusing Large Biomedical Knowledge Graphs into BERT
Infusing factual knowledge into pre-trained models is fundamental for many knowledge-intensive tasks. In this paper, we proposed Mixture-of-Partitions (MoP), an infusion approach that can handle a very large knowledge gr…
Knowledge Graphs