Pre-training Transformers for Knowledge Graph Completion
Learning transferable representation of knowledge graphs (KGs) is challenging due to the heterogeneous, multi-relational nature of graph structures. Inspired by Transformer-based pretrained language models' success on learning transferable representation for texts, we introduce a novel inductive KG representation model (iHT) for KG completion by large-scale pre-training. iHT consists of a entity encoder (e.g., BERT) and a neighbor-aware relational scoring function both parameterized by Transformers. We first pre-train iHT on a large KG dataset, Wikidata5M. Our approach achieves new state-of-the-art results on matched evaluations, with a relative improvement of more than 25% in mean reciprocal rank over previous SOTA models. When further fine-tuned on smaller KGs with either entity and relational shifts, pre-trained iHT representations are shown to be transferable, significantly improving the performance on FB15K-237 and WN18RR.
Code (0)
등록된 구현이 없습니다.
Tasks
Knowledge Graph CompletionKnowledge GraphsSimilar Papers 제목 키워드 기반
Using Graph Algorithms to Pretrain Graph Completion Transformers
Recent work on Graph Neural Networks has demonstrated that self-supervised pretraining can further enhance performance on downstream graph, link, and node classification tasks. However, the efficacy of pretraining tasks …
Graph EmbeddingKnowledge Graph CompletionKnowledge Graph EmbeddingNode ClassificationCOMET: Commonsense Transformers for Automatic Knowledge Graph Construction
We present the first comprehensive study on automatic knowledge base construction for two prevalent commonsense knowledge graphs: ATOMIC (Sap et al., 2019) and ConceptNet (Speer et al., 2017). Contrary to many convention…
graph constructionKnowledge Base ConstructionKnowledge GraphsA Simple But Powerful Graph Encoder for Temporal Knowledge Graph Completion
While knowledge graphs contain rich semantic knowledge about various entities and the relational information among them, temporal knowledge graphs (TKGs) describe and model the interactions of the entities over time. In …
Knowledge Graph CompletionKnowledge GraphsTemporal Knowledge Graph CompletionA Simple But Powerful Graph Encoder for Temporal Knowledge Graph Completion
Knowledge graphs contain rich knowledge about various entities and the relational information among them, while temporal knowledge graphs (TKGs) describe and model the interactions of the entities over time. In this cont…
Knowledge Graph CompletionKnowledge GraphsTemporal Knowledge Graph CompletionA Pre-training Framework for Knowledge Graph Completion
Knowledge graph completion (KGC) is one of the effective methods to identify new facts in knowledge graph. Except for a few methods based on graph network, most of KGC methods trend to be trained based on independent tri…
Knowledge Graph Completion