Rethinking Graph Transformer Architecture Design for Node Classification
Graph Transformer (GT), as a special type of Graph Neural Networks (GNNs), utilizes multi-head attention to facilitate high-order message passing. However, this also imposes several limitations in node classification applications: 1) nodes are susceptible to global noise; 2) self-attention computation cannot scale well to large graphs. In this work, we conduct extensive observational experiments to explore the adaptability of the GT architecture in node classification tasks and draw several conclusions: the current multi-head self-attention module in GT can be completely replaceable, while the feed-forward neural network module proves to be valuable. Based on this, we decouple the propagation (P) and transformation (T) of GNNs and explore a powerful GT architecture, named GNNFormer, which is based on the P/T combination message passing and adapted for node classification in both homophilous and heterophilous scenarios. Extensive experiments on 12 benchmark datasets demonstrate that our proposed GT architecture can effectively adapt to node classification tasks without being affected by global noise and computational efficiency limitations.
Code (0)
등록된 구현이 없습니다.
Tasks
ClassificationComputational EfficiencyNode ClassificationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
NN-Former: Rethinking Graph Structure in Neural Architecture Representation
The growing use of deep learning necessitates efficient network design and deployment, making neural predictors vital for estimating attributes such as accuracy and latency. Recently, Graph Neural Networks (GNNs) and tra…
NN-Former: Rethinking Graph Structure in Neural Architecture Representation
The growing use of deep learning necessitates efficient network design and deployment, making neural predictors vital for estimating attributes such as accuracy and latency. Recently, Graph Neural Networks (GNNs) and…
Rethinking Graph Transformers with Spectral Attention
In recent years, the Transformer architecture has proven to be very successful in sequence processing, but its application to other data structures, such as graphs, has remained limited due to the difficulty of properly …
Rethinking Tokenized Graph Transformers for Node Classification
Node tokenized graph Transformers (GTs) have shown promising performance in node classification. The generation of token sequences is the key module in existing tokenized GTs which transforms the input graph into token s…
ClassificationNode ClassificationRepresentation LearningGraph Propagation Transformer for Graph Representation Learning
This paper presents a novel transformer architecture for graph representation learning. The core insight of our method is to fully consider the information propagation among nodes and edges in a graph when building the a…
Graph LearningGraph Property PredictionGraph RegressionGraph Representation Learning+2