BERT-MK: Integrating Graph Contextualized Knowledge into Pre-trained Language Models
Complex node interactions are common in knowledge graphs (KGs), and these interactions can be considered as contextualized knowledge exists in the topological structure of KGs. Traditional knowledge representation learning (KRL) methods usually treat a single triple as a training unit, neglecting the usage of graph contextualized knowledge. To utilize these unexploited graph-level knowledge, we propose an approach to model subgraphs in a medical KG. Then, the learned knowledge is integrated with a pre-trained language model to do the knowledge generalization. Experimental results demonstrate that our model achieves the state-of-the-art performance on several medical NLP tasks, and the improvement above MedERNIE indicates that graph contextualized knowledge is beneficial.
Code (0)
등록된 구현이 없습니다.
Tasks
Knowledge GraphsLanguage ModelingLanguage ModellingRepresentation LearningSimilar Papers 제목 키워드 기반
Integrating Graph Contextualized Knowledge into Pre-trained Language Models
Complex node interactions are common in knowledge graphs, and these interactions also contain rich knowledge information. However, traditional methods usually treat a triple as a training unit during the knowledge repres…
Knowledge GraphsRepresentation LearningCoLAKE: Contextualized Language and Knowledge Embedding
With the emerging branch of incorporating factual knowledge into pre-trained language models such as BERT, most existing models consider shallow, static, and separately pre-trained entity embeddings, which limits the per…
Entity EmbeddingsKnowledge Graph CompletionKnowledge ProbingKI-BERT: Infusing Knowledge Context for Better Language and Domain Understanding
Contextualized entity representations learned by state-of-the-art transformer-based language models (TLMs) like BERT, GPT, T5, etc., leverage the attention mechanism to learn the data context from training data corpus. H…
Graph EmbeddingKnowledge Graph EmbeddingKnowledge Graph EmbeddingsKnowledge Graphs+2PAI at SemEval-2022 Task 11: Name Entity Recognition with Contextualized Entity Representations and Robust Loss Functions
This paper describes our system used in the SemEval-2022 Task 11 Multilingual Complex Named Entity Recognition, achieving 3rd for track 1 on the leaderboard. We propose Dictionary-fused BERT, a flexible approach for enti…
Binary Classificationnamed-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)Cross-Modal Contextualized Hidden State Projection Method for Expanding of Taxonomic Graphs
Taxonomy is a graph of terms organized hierarchically using is-a (hypernymy) relations. We suggest novel candidate-free task formulation for the taxonomy enrichment task. To solve the task, we leverage lexical knowledge …
graph constructionGraph Embedding