Improving Knowledge Graph Representation Learning by Structure Contextual Pre-training

2021-12-08 · Ganqiang Ye, Wen Zhang, Zhen Bi, Chi Man Wong, Chen Hui, Huajun Chen

Representation learning models for Knowledge Graphs (KG) have proven to be effective in encoding structural information and performing reasoning over KGs. In this paper, we propose a novel pre-training-then-fine-tuning framework for knowledge graph representation learning, in which a KG model is firstly pre-trained with triple classification task, followed by discriminative fine-tuning on specific downstream tasks such as entity type prediction and entity alignment. Drawing on the general ideas of learning deep contextualized word representations in typical pre-trained language models, we propose SCoP to learn pre-trained KG representations with structural and contextual triples of the target triple encoded. Experimental results demonstrate that fine-tuning SCoP not only outperforms results of baselines on a portfolio of downstream tasks but also avoids tedious task-specific model design and parameter training.

📄 PDF Abstract BibTeX arXiv:2112.04087

Code (0)

등록된 구현이 없습니다.

Tasks

Entity Alignment Graph Representation Learning Knowledge Graphs Representation Learning Triple Classification Type prediction

Methods 이 논문이 사용한 방법론

Discriminative Fine-Tuning Discriminative Fine-Tuning is a fine-tuning strategy that is used for ULMFiT type models. Instead of using the same learning rate…

PPKE: Knowledge Representation Learning by Path-based Pre-training

2020-12-07 · Bin He, Di Zhou, Jing Xie, Jinghui Xiao 외

Entities may have complex interactions in a knowledge graph (KG), such as multi-step relationships, which can be viewed as graph contextual information of the entities. Traditional knowledge representation learning (KRL)…

Link Prediction Relation Prediction Representation Learning

BERT-MK: Integrating Graph Contextualized Knowledge into Pre-trained Language Models

2020-11-01 · Findings of the Association for Computational Linguistics 2020 · Bin He, Di Zhou, Jinghui Xiao, Xin Jiang 외

Complex node interactions are common in knowledge graphs (KGs), and these interactions can be considered as contextualized knowledge exists in the topological structure of KGs. Traditional knowledge representation learni…

Knowledge Graphs Language Modeling Language Modelling Representation Learning

Explicit Graph Reasoning Fusing Knowledge and Contextual Information for Multi-hop Question Answering

2022-07-01 · NAACL (DLG4NLP) 2022 7 · Zhenyun Deng, Yonghua Zhu, Qianqian Qi, Michael Witbrock 외

Current graph-neural-network-based (GNN-based) approaches to multi-hop questions integrate clues from scattered paragraphs in an entity graph, achieving implicit reasoning by synchronous update of graph node representati…

Graph Neural Network Multi-hop Question Answering Question Answering Sentence

CoLAKE: Contextualized Language and Knowledge Embedding

2020-10-01 · COLING 2020 8 · Tianxiang Sun, Yunfan Shao, Xipeng Qiu, Qipeng Guo 외

With the emerging branch of incorporating factual knowledge into pre-trained language models such as BERT, most existing models consider shallow, static, and separately pre-trained entity embeddings, which limits the per…

Entity Embeddings Knowledge Graph Completion Knowledge Probing

KG-CQR: Leveraging Structured Relation Representations in Knowledge Graphs for Contextual Query Retrieval

2025-08-28 · Chi Minh Bui, Ngoc Mai Thieu, Van Vinh Nguyen, Jason J. Jung 외 arxiv

The integration of knowledge graphs (KGs) with large language models (LLMs) offers significant potential to improve the retrieval phase of retrieval-augmented generation (RAG) systems. In this study, we propose KG-CQR, a…

Multi-hop Question Answering Knowledge Graphs