KLMo: Knowledge Graph Enhanced Pretrained Language Model with Fine-Grained Relationships
Interactions between entities in knowledge graph (KG) provide rich knowledge for language representation learning. However, existing knowledge-enhanced pretrained language models (PLMs) only focus on entity information and ignore the fine-grained relationships between entities. In this work, we propose to incorporate KG (including both entities and relations) into the language learning process to obtain KG-enhanced pretrained Language Model, namely KLMo. Specifically, a novel knowledge aggregator is designed to explicitly model the interaction between entity spans in text and all entities and relations in a contextual KG. An relation prediction objective is utilized to incorporate relation information by distant supervision. An entity linking objective is further utilized to link entity spans in text to entities in KG. In this way, the structured knowledge can be effectively integrated into language representations. Experimental results demonstrate that KLMo achieves great improvements on several knowledge-driven tasks, such as entity typing and relation classification, comparing with the state-of-the-art knowledge-enhanced PLMs.
Code (1)
Tasks
Entity LinkingEntity TypingLanguage ModelingLanguage ModellingRelationRelation ClassificationRelation PredictionRepresentation LearningSimilar Papers 제목 키워드 기반
Commonsense Knowledge-Augmented Pretrained Language Models for Causal Reasoning Classification
Commonsense knowledge can be leveraged for identifying causal relations in text. In this work, we convert triples in ATOMIC2020, a wide coverage commonsense reasoning knowledge graph, to natural language text and continu…
ClassificationCommonsense Causal ReasoningLanguage ModelingLanguage ModellingWhat Has Been Enhanced in my Knowledge-Enhanced Language Model?
Pretrained language models (LMs) do not capture factual knowledge very well. This has led to the development of a number of knowledge integration (KI) methods which aim to incorporate external knowledge into pretrained L…
Graph AttentionLanguage ModelingLanguage ModellingKnowledge-Augmented Language Models for Cause-Effect Relation Classification
Previous studies have shown the efficacy of knowledge augmentation methods in pretrained language models. However, these methods behave differently across domains and downstream tasks. In this work, we investigate the au…
Cause-Effect Relation ClassificationClassificationCommonsense Causal ReasoningLanguage Modelling+2GreaseLM: Graph REASoning Enhanced Language Models
Answering complex questions about textual narratives requires reasoning over both stated context and the world knowledge that underlies it. However, pretrained language models (LM), the foundation of most modern QA syste…
Knowledge GraphsMedical Question AnsweringMedQANegation+3GreaseLM: Graph REASoning Enhanced Language Models for Question Answering
Answering complex questions about textual narratives requires reasoning over both stated context and the world knowledge that underlies it. However, pretrained language models (LM), the foundation of most modern QA syste…
Knowledge GraphsMedical Question AnsweringMedQANegation+3