Pure Transformers are Powerful Graph Learners

2022-07-06 · Jinwoo Kim, Tien Dat Nguyen, Seonwoo Min, Sungjun Cho, Moontae Lee, Honglak Lee, Seunghoon Hong

We show that standard Transformers without graph-specific modifications can lead to promising results in graph learning both in theory and practice. Given a graph, we simply treat all nodes and edges as independent tokens, augment them with token embeddings, and feed them to a Transformer. With an appropriate choice of token embeddings, we prove that this approach is theoretically at least as expressive as an invariant graph network (2-IGN) composed of equivariant linear layers, which is already more expressive than all message-passing Graph Neural Networks (GNN). When trained on a large-scale graph dataset (PCQM4Mv2), our method coined Tokenized Graph Transformer (TokenGT) achieves significantly better results compared to GNN baselines and competitive results compared to Transformer variants with sophisticated graph-specific inductive bias. Our implementation is available at https://github.com/jw9730/tokengt.

📄 PDF Abstract BibTeX arXiv:2207.02505

Code (2)

jw9730/tokengt 공식 구현 pytorch

luis-mueller/wl-transformers pytorch

Tasks

Graph Classification Graph Learning Graph RegressionInductive BiasMolecular Property Prediction

Methods 이 논문이 사용한 방법론

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

LapEigen 설명 없음

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Multi-Head Attention 설명 없음

Residual Connection 설명 없음

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Position-Wise Feed-Forward Layer 설명 없음

Plain Transformers Can be Powerful Graph Learners

2025-04-17 · Liheng Ma, Soumyasundar Pal, Yingxue Zhang, Philip H. S. Torr 외

Transformers have attained outstanding performance across various modalities, owing to their simple but powerful scaled-dot-product (SDP) attention mechanisms. Researchers have attempted to migrate Transformers to graph …

Graph Learning

When BERT Fails -- The Limits of EHR Classification

2022-07-26 · Augusto Garcia-Agundez, Carsten Eickhoff

Transformers are powerful text representation learners, useful for all kinds of clinical decision support tasks. Although they outperform baselines on readmission prediction, they are not infallible. Here, we look into o…

Classification Readmission Prediction

Online Finetuning Decision Transformers with Pure RL Gradients

2026-01-01 · Junkai Luo, Yinglun Zhu arxiv

Decision Transformers (DTs) have emerged as a powerful framework for sequential decision making by formulating offline reinforcement learning (RL) as a sequence modeling problem. However, extending DTs to online settings…

Reinforcement Learning Decision Making

SPACE: Your Genomic Profile Predictor is a Powerful DNA Foundation Model

2025-06-02 · Zhao Yang, Jiwei Zhu, Bing Su

Inspired by the success of unsupervised pre-training paradigms, researchers have applied these approaches to DNA pre-training. However, we argue that these approaches alone yield suboptimal results because pure DNA seque…

Mixture-of-Experts Unsupervised Pre-training

Transformers are efficient hierarchical chemical graph learners

2023-10-02 · Zihan Pengmei, Zimu Li, Chih-chan Tien, Risi Kondor 외

Transformers, adapted from natural language processing, are emerging as a leading approach for graph representation learning. Contemporary graph transformers often treat nodes or edges as separate tokens. This approach l…

Graph Representation Learning Representation Learning