Injecting Knowledge into Biomedical Pre-trained Models via Polymorphism and Synonymous Substitution
Pre-trained language models (PLMs) were considered to be able to store relational knowledge present in the training data. However, some relational knowledge seems to be discarded unsafely in PLMs due to \textbf{report bias}: low-frequency relational knowledge might be underexpressed compared to high-frequency one in PLMs. This gives us a hint that relational knowledge might not be redundant to the stored knowledge of PLMs, but rather be complementary. To additionally inject relational knowledge into PLMs, we propose a simple-yet-effective approach to inject relational knowledge into PLMs, which is inspired by three observations (namely, polymorphism, synonymous substitution, and association). In particular, we switch entities in the training corpus to related entities (either hypernyms/hyponyms/synonyms, or arbitrarily-related concepts). Experimental results show that the proposed approach could not only better capture relational knowledge, but also improve the performance in various biomedical downstream tasks. Our model is available in \url{https://github.com/StevenZHB/BioPLM_InjectingKnowledge}.
Code (1)
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Biomedical Entity Linking with Triple-aware Pre-Training
Linking biomedical entities is an essential aspect in biomedical natural language processing tasks, such as text mining and question answering. However, a difficulty of linking the biomedical entities using current large…
Entity LinkingQuestion AnsweringLearning Domain-Specialised Representations for Cross-Lingual Biomedical Entity Linking
Injecting external domain-specific knowledge (e.g., UMLS) into pretrained language models (LMs) advances their capability to handle specialised in-domain tasks such as biomedical entity linking (BEL). However, such abund…
Cross-Lingual TransferEntity LinkingFrom Knowledge to Treatment: Large Language Model Assisted Biomedical Concept Representation for Drug Repurposing
Drug repurposing plays a critical role in accelerating treatment discovery, especially for complex and rare diseases. Biomedical knowledge graphs (KGs), which encode rich clinical associations, have been widely adopted t…
Knowledge Graph EmbeddingKnowledge GraphsInjecting Structured Biomedical Knowledge into Language Models: Continual Pretraining vs. GraphRAG
The injection of domain-specific knowledge is crucial for adapting language models (LMs) to specialized fields such as biomedicine. While most current approaches rely on unstructured text corpora, this study explores two…
Continual PretrainingQuestion AnsweringCausalBERT: Injecting Causal Knowledge Into Pre-trained Models with Minimal Supervision
Recent work has shown success in incorporating pre-trained models like BERT to improve NLP systems. However, existing pre-trained models lack of causal knowledge which prevents today's NLP systems from thinking like huma…
Causal Inference