Bridging Graph Position Encodings for Transformers with Weighted Graph-Walking Automata
A current goal in the graph neural network literature is to enable transformers to operate on graph-structured data, given their success on language and vision tasks. Since the transformer's original sinusoidal positional encodings (PEs) are not applicable to graphs, recent work has focused on developing graph PEs, rooted in spectral graph theory or various spatial features of a graph. In this work, we introduce a new graph PE, Graph Automaton PE (GAPE), based on weighted graph-walking automata (a novel extension of graph-walking automata). We compare the performance of GAPE with other PE schemes on both machine translation and graph-structured tasks, and we show that it generalizes several other PEs. An additional contribution of this study is a theoretical and controlled experimental comparison of many recent PEs in graph transformers, independent of the use of edge features.
Code (0)
등록된 구현이 없습니다.
Tasks
Graph Neural NetworkMachine TranslationPositionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Comparing Graph Transformers via Positional Encodings
The distinguishing power of graph transformers is closely tied to the choice of positional encoding: features used to augment the base transformer with information about the graph. There are two primary types of position…
NavigateHyPE-GT: where Graph Transformers meet Hyperbolic Positional Encodings
Graph Transformers (GTs) facilitate the comprehension of graph-structured data by calculating the self-attention of node pairs without considering node position information. To address this limitation, we introduce an in…
PositionGraph Transformers without Positional Encodings
Recently, Transformers for graph representation learning have become increasingly popular, achieving state-of-the-art performance on a wide-variety of graph datasets, either alone or in combination with message-passing g…
Graph ClassificationGraph RegressionGraph Representation LearningNode Classification+1Size Transferability of Graph Transformers with Convolutional Positional Encodings
Transformers have achieved remarkable success across domains, motivating the rise of Graph Transformers (GTs) as attention-based architectures for graph-structured data. A key design choice in GTs is the use of Graph Neu…
Graph Neural NetworkBridging Distance and Spectral Positional Encodings via Anchor-Based Diffusion Geometry Approximation
Molecular graph learning benefits from positional signals that capture both local neighborhoods and global topology. Two widely used families are spectral encodings derived from Laplacian or diffusion operators and ancho…
Graph Learning