T-Graphormer: Using Transformers for Spatiotemporal Forecasting

2025-01-22 · Hao Yuan Bai, Xue Liu

Spatiotemporal data is ubiquitous, and forecasting it has important applications in many domains. However, its complex cross-component dependencies and non-linear temporal dynamics can be challenging for traditional techniques. Existing methods address this by learning the two dimensions separately. Here, we introduce Temporal Graphormer (T-Graphormer), a Transformer-based approach capable of modelling spatiotemporal correlations simultaneously. By adding temporal encodings in the Graphormer architecture, each node attends to all other tokens within the graph sequence, enabling the model to learn rich spacetime patterns with minimal predefined inductive biases. We show the effectiveness of T-Graphormer on real-world traffic prediction benchmark datasets. Compared to state-of-the-art methods, T-Graphormer reduces root mean squared error (RMSE) and mean absolute percentage error (MAPE) by up to 20% and 10%.

📄 PDF Abstract BibTeX arXiv:2501.13274

Code (1)

rdh1115/T-Graphormer 공식 구현 pytorch

Tasks

Time SeriesTraffic Prediction

Methods 이 논문이 사용한 방법론

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

Position-Wise Feed-Forward Layer 설명 없음

Residual Connection 설명 없음

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Adam 설명 없음

Do Transformers Really Perform Badly for Graph Representation?

2021-12-01 · NeurIPS 2021 12 · Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng 외

The Transformer architecture has become a dominant choice in many domains, such as natural language processing and computer vision. Yet, it has not achieved competitive performance on popular leaderboards of graph-level …

Graph Representation Learning Representation Learning

Do Transformers Really Perform Bad for Graph Representation?

2021-06-09 · Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng 외

Graph Classification Graph Property Prediction Graph Regression Graph Representation Learning+2

SpikeGraphormer: A High-Performance Graph Transformer with Spiking Graph Attention

2024-03-21 · Yundong Sun, Dongjie Zhu, Yansong Wang, Zhaoshuo Tian 외

Recently, Graph Transformers have emerged as a promising solution to alleviate the inherent limitations of Graph Neural Networks (GNNs) and enhance graph representation performance. Unfortunately, Graph Transformers are …

GPU Graph Attention text-classification Text Classification

Mesh Graphormer

2021-04-01 · ICCV 2021 10 · Kevin Lin, Lijuan Wang, Zicheng Liu

We present a graph-convolution-reinforced transformer, named Mesh Graphormer, for 3D human pose and mesh reconstruction from a single image. Recently both transformers and graph convolutional neural networks (GCNNs) have…

3D Hand Pose Estimation 3D Human Pose Estimation

Long-Range Transformers for Dynamic Spatiotemporal Forecasting

2021-09-24 · Jake Grigsby, Zhe Wang, Nam Nguyen, Yanjun Qi

Multivariate time series forecasting focuses on predicting future values based on historical context. State-of-the-art sequence-to-sequence models rely on neural attention between timesteps, which allows for temporal lea…

Multivariate Time Series Forecasting Time Series Time Series Analysis Time Series Forecasting