Time-Delayed Transformers for Data-Driven Modeling of Low-Dimensional Dynamics
We propose the time-delayed transformer (TD-TF), a simplified transformer architecture for data-driven modeling of unsteady spatio-temporal dynamics. TD-TF bridges linear operator-based methods and deep sequence models by showing that a single-layer, single-head transformer can be interpreted as a nonlinear generalization of time-delayed dynamic mode decomposition (TD-DMD). The architecture is deliberately minimal, consisting of one self-attention layer with a single query per prediction and one feedforward layer, resulting in linear computational complexity in sequence length and a small parameter count. Numerical experiments demonstrate that TD-TF matches the performance of strong linear baselines on near-linear systems, while significantly outperforming them in nonlinear and chaotic regimes, where it accurately captures long-term dynamics. Validation studies on synthetic signals, unsteady aerodynamics, the Lorenz '63 system, and a reaction-diffusion model show that TD-TF preserves the interpretability and efficiency of linear models while providing substantially enhanced expressive power for complex dynamics.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
CSDformer: A Conversion Method for Fully Spike-Driven Transformer
Spike-based transformer is a novel architecture aiming to enhance the performance of spiking neural networks while mitigating the energy overhead inherent to transformers. However, methods for generating these models suf…
A Bayesian Perspective on the Role of Epistemic Uncertainty for Delayed Generalization in In-Context Learning
In-context learning enables transformers to adapt to new tasks from a few examples at inference time, while grokking highlights that this generalization can emerge abruptly only after prolonged training. We study task ge…
Analysis, forecasting and system identification of a floating offshore wind turbine using dynamic mode decomposition
This article presents the data-driven equation-free modeling of the dynamics of a hexafloat floating offshore wind turbine based on the application of dynamic mode decomposition (DMD). All the analyses are performed on e…
Uncertainty QuantificationTraining-Free Acceleration of ViTs with Delayed Spatial Merging
Token merging has emerged as a new paradigm that can accelerate the inference of Vision Transformers (ViTs) without any retraining or fine-tuning. To push the frontier of training-free acceleration in ViTs, we improve to…
Transfer LearningStreaming Sequence-to-Sequence Learning with Delayed Streams Modeling
We introduce Delayed Streams Modeling (DSM), a flexible formulation for streaming, multimodal sequence-to-sequence learning. Sequence-to-sequence generation is often cast in an offline manner, where the model consumes th…
Speech Recognition