GraphGPT: Graph Learning with Generative Pre-trained Transformers
We introduce \textit{GraphGPT}, a novel model for Graph learning by self-supervised Generative Pre-training Transformers. Our model transforms each graph or sampled subgraph into a sequence of tokens representing the node, edge and attributes reversibly using the Eulerian path first. Then we feed the tokens into a standard transformer decoder and pre-train it with the next-token-prediction (NTP) task. Lastly, we fine-tune the GraphGPT model with the supervised tasks. This intuitive, yet effective model achieves superior or close results to the state-of-the-art methods for the graph-, edge- and node-level tasks on the large scale molecular dataset PCQM4Mv2, the protein-protein association dataset ogbl-ppa and the ogbn-proteins dataset from the Open Graph Benchmark (OGB). Furthermore, the generative pre-training enables us to train GraphGPT up to 400M+ parameters with consistently increasing performance, which is beyond the capability of GNNs and previous graph transformers. The source code and pre-trained checkpoints will be released soon\footnote{\url{https://github.com/alibaba/graph-gpt}} to pave the way for the graph foundation model research, and also to assist the scientific discovery in pharmaceutical, chemistry, material and bio-informatics domains, etc.
Code (1)
Tasks
DecoderGraph Learningscientific discoverySimilar Papers 제목 키워드 기반
A Graph is Worth $K$ Words: Euclideanizing Graph using Pure Transformer
Can we model Non-Euclidean graphs as pure language or even Euclidean vectors while retaining their inherent information? The Non-Euclidean property have posed a long term challenge in graph modeling. Despite recent graph…
DecoderGraph ClassificationGraph GenerationGraph Representation Learning+1GraphGPT: Graph Instruction Tuning for Large Language Models
Graph Neural Networks (GNNs) have evolved to understand graph structures through recursive exchanges and aggregations among nodes. To enhance robustness, self-supervised learning (SSL) has become a vital tool for data au…
Data AugmentationGraph LearningSelf-Supervised LearningZero-Shot LearningGRAPHGPT-O: Synergistic Multimodal Comprehension and Generation on Graphs
The rapid development of Multimodal Large Language Models (MLLMs) has enabled the integration of multiple modalities, including texts and images, within the large language model (LLM) framework. However, texts and images…
Image GenerationLanguage ModelingLanguage ModellingLarge Language ModelCustomized Information and Domain-centric Knowledge Graph Construction with Large Language Models
In this paper we propose a novel approach based on knowledge graphs to provide timely access to structured information, to enable actionable technology intelligence, and improve cyber-physical systems planning. Our frame…
Decision Makinggraph constructionInformation RetrievalKeyphrase Extraction+1Scaling Generative Foundation Models for Chest Radiography with Rectified Flow Transformers
We introduce the first generative foundation model for chest radiograph synthesis trained from scratch at the billion-parameter scale. Existing radiographic AI models often suffer from poor generalisation across patient …