LPNL: Scalable Link Prediction with Large Language Models

2024-01-24 · Baolong Bi, Shenghua Liu, Yiwei Wang, Lingrui Mei, Xueqi Cheng

Exploring the application of large language models (LLMs) to graph learning is a emerging endeavor. However, the vast amount of information inherent in large graphs poses significant challenges to this process. This work focuses on the link prediction task and introduces $\textbf{LPNL}$ (Link Prediction via Natural Language), a framework based on large language models designed for scalable link prediction on large-scale heterogeneous graphs. We design novel prompts for link prediction that articulate graph details in natural language. We propose a two-stage sampling pipeline to extract crucial information from the graphs, and a divide-and-conquer strategy to control the input tokens within predefined limits, addressing the challenge of overwhelming information. We fine-tune a T5 model based on our self-supervised learning designed for link prediction. Extensive experimental results demonstrate that LPNL outperforms multiple advanced baselines in link prediction tasks on large-scale graphs.

📄 PDF Abstract BibTeX arXiv:2401.13227

Code (0)

등록된 구현이 없습니다.

Tasks

Graph Learning Language ModellingLarge Language ModelLink Prediction Prediction Self-Supervised Learning

Methods 이 논문이 사용한 방법론

Gated Linear Unit A Gated Linear Unit, or GLU computes: $$ \mathrm{GLU}(a, b) = a \otimes \sigma(b) $$ It is used in natural language processing architectures, for example the Gated CNN,…

Attention 설명 없음

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Residual Connection 설명 없음

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

SentencePiece 설명 없음

Inverse Square Root Schedule Inverse Square Root is a learning rate schedule 1 / $\sqrt{\max\left(n, k\right)}$ where $n$ is the current training iteration and $k$ is the number of warm-up steps. This…

Adafactor Adafactor is a stochastic optimization method based on Adam that reduces memory usage while retaining the empirical benefits of…

Stylistic MR-to-Text Generation Using Pre-trained Language Models

2021-12-01 · ICON 2021 12 · Kunal Pagarey, Kanika Kalra, Abhay Garg, Saumajit Saha 외

We explore the ability of pre-trained language models BART, an encoder-decoder model, GPT2 and GPT-Neo, both decoder-only models for generating sentences from structured MR tags as input. We observe best results on sever…

Decoder POS Sentence Text Generation

Simplifying Subgraph Representation Learning for Scalable Link Prediction

2023-01-29 · Paul Louis, Shweta Ann Jacob, Amirali Salehi-Abari

Link prediction on graphs is a fundamental problem. Subgraph representation learning approaches (SGRLs), by transforming link prediction to graph classification on the subgraphs around the links, have achieved state-of-t…

Graph Classification Graph Learning Graph Representation Learning Link Prediction+2

Scalable and interpretable rule-based link prediction for large heterogeneous knowledge graphs

2020-12-10 · Simon Ott, Laura Graf, Asan Agibetov, Christian Meilicke 외

Neural embedding-based machine learning models have shown promise for predicting novel links in biomedical knowledge graphs. Unfortunately, their practical utility is diminished by their lack of interpretability. Recentl…

Clustering Knowledge Graphs Link Prediction Prediction

Hierarchical Hyperlink Prediction for the WWW

2016-11-28 · Garcia-Gasulla Dario, Ayguadé Eduard, Labarta Jesús, Cortés Ulises 외

The hyperlink prediction task, that of proposing new links between webpages, can be used to improve search engines, expand the visibility of web pages, and increase the connectivity and navigability of the web. Hyperlink…

Link Prediction Prediction

Scalable Link Prediction in Dynamic Networks via Non-Negative Matrix Factorization

2014-11-13 · Linhong Zhu, Dong Guo, Junming Yin, Greg Ver Steeg 외

We propose a scalable temporal latent space model for link prediction in dynamic social networks, where the goal is to predict links over time based on a sequence of previous graph snapshots. The model assumes that each …

global-optimization Link Prediction Prediction