Continuous Spatiotemporal Transformers

2023-01-31 · Antonio H. de O. Fonseca, Emanuele Zappala, Josue Ortega Caro, David van Dijk

Modeling spatiotemporal dynamical systems is a fundamental challenge in machine learning. Transformer models have been very successful in NLP and computer vision where they provide interpretable representations of data. However, a limitation of transformers in modeling continuous dynamical systems is that they are fundamentally discrete time and space models and thus have no guarantees regarding continuous sampling. To address this challenge, we present the Continuous Spatiotemporal Transformer (CST), a new transformer architecture that is designed for the modeling of continuous systems. This new framework guarantees a continuous and smooth output via optimization in Sobolev space. We benchmark CST against traditional transformers as well as other spatiotemporal dynamics modeling methods and achieve superior performance in a number of tasks on synthetic and real systems, including learning brain dynamics from calcium imaging data.

📄 PDF Abstract BibTeX arXiv:2301.13338

Code (1)

vandijklab/cst 공식 구현 pytorch

Methods 이 논문이 사용한 방법론

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

Adam 설명 없음

Multi-Head Attention 설명 없음

Residual Connection 설명 없음

CardioDiT: Latent Diffusion Transformers for 4D Cardiac MRI Synthesis

2026-03-26 · Marvin Seyfarth, Sarah Kaye Müller, Arman Ghanaat, Isabelle Ayx 외 arxiv

Latent diffusion models (LDMs) have recently achieved strong performance in 3D medical image synthesis. However, modalities like cine cardiac MRI (CMR), representing a temporally synchronized 3D volume across the cardiac…

TiMo: Spatiotemporal Foundation Model for Satellite Image Time Series

2025-05-13 · Xiaolei Qin, Di Wang, Jing Zhang, Fengxiang Wang 외

Satellite image time series (SITS) provide continuous observations of the Earth's surface, making them essential for applications such as environmental management and disaster assessment. However, existing spatiotemporal…

Temporal Sequences Time Series

SSTVOS: Sparse Spatiotemporal Transformers for Video Object Segmentation

2021-01-21 · CVPR 2021 1 · Brendan Duke, Abdalla Ahmed, Christian Wolf, Parham Aarabi 외

In this paper we introduce a Transformer-based approach to video object segmentation (VOS). To address compounding error and scalability issues of prior work, we propose a scalable, end-to-end method for VOS called Spars…

Inductive Bias Motion Segmentation Object One-shot visual object segmentation+6

Video Prediction of Dynamic Physical Simulations With Pixel-Space Spatiotemporal Transformers

2025-10-23 · Dean L Slack, G Thomas Hudson, Thomas Winterbottom, Noura Al Moubayed arxiv

Inspired by the performance and scalability of autoregressive large language models (LLMs), transformer-based models have seen recent success in the visual domain. This study investigates a transformer adaptation for vid…

Physical Simulations Video Prediction Object Tracking

Spatiotemporal Self-attention Modeling with Temporal Patch Shift for Action Recognition

2022-07-27 · Wangmeng Xiang, Chao Li, Biao Wang, Xihan Wei 외

Transformer-based methods have recently achieved great advancement on 2D image-based vision tasks. For 3D video-based tasks such as action recognition, however, directly applying spatiotemporal transformers on video data…

Action Classification Action Recognition