Attention as Robust Representation for Time Series Forecasting

2024-02-08 · Peisong Niu, Tian Zhou, Xue Wang, Liang Sun, Rong Jin

Time series forecasting is essential for many practical applications, with the adoption of transformer-based models on the rise due to their impressive performance in NLP and CV. Transformers' key feature, the attention mechanism, dynamically fusing embeddings to enhance data representation, often relegating attention weights to a byproduct role. Yet, time series data, characterized by noise and non-stationarity, poses significant forecasting challenges. Our approach elevates attention weights as the primary representation for time series, capitalizing on the temporal relationships among data points to improve forecasting accuracy. Our study shows that an attention map, structured using global landmarks and local windows, acts as a robust kernel representation for data points, withstanding noise and shifts in distribution. Our method outperforms state-of-the-art models, reducing mean squared error (MSE) in multivariate time series forecasting by a notable 3.6% without altering the core neural network architecture. It serves as a versatile component that can readily replace recent patching based embedding schemes in transformer-based models, boosting their performance.

📄 PDF Abstract BibTeX arXiv:2402.05370

Code (0)

등록된 구현이 없습니다.

Tasks

Multivariate Time Series ForecastingTime SeriesTime Series Forecasting

Methods 이 논문이 사용한 방법론

Patching Activation patching studies the model's computation by altering its latent representations, the token embeddings in transformer-based language models, during the inference process

Expressing Multivariate Time Series as Graphs with Time Series Attention Transformer

2022-08-19 · William T. Ng, K. Siu, Albert C. Cheung, Michael K. Ng

A reliable and efficient representation of multivariate time series is crucial in various downstream machine learning tasks. In multivariate time series forecasting, each variable depends on its historical values and the…

Multivariate Time Series Forecasting Representation Learning Time Series Time Series Analysis+1

FlexTSF: A Universal Forecasting Model for Time Series with Variable Regularities

2024-10-30 · Jingge Xiao, Yile Chen, Gao Cong, Wolfgang Nejdl 외

Developing a foundation model for time series forecasting across diverse domains has attracted significant attention in recent years. Existing works typically assume regularly sampled, well-structured data, limiting thei…

Irregular Time Series Missing Values Time Series Time Series Forecasting

DRAformer: Differentially Reconstructed Attention Transformer for Time-Series Forecasting

2022-06-11 · Benhan Li, Shengdong Du, Tianrui Li, Jie Hu 외

Time-series forecasting plays an important role in many real-world scenarios, such as equipment life cycle forecasting, weather forecasting, and traffic flow forecasting. It can be observed from recent research that a va…

Decoder Time Series Time Series Analysis Time Series Forecasting+1

Temporal-Spatial dependencies ENhanced deep learning model (TSEN) for household leverage series forecasting

2022-10-17 · Hu Yang, Yi Huang, Haijun Wang, Yu Chen

Analyzing both temporal and spatial patterns for an accurate forecasting model for financial time series forecasting is a challenge due to the complex nature of temporal-spatial dynamics: time series from different locat…

Time Series Time Series Analysis Time Series Forecasting

S2TX: Cross-Attention Multi-Scale State-Space Transformer for Time Series Forecasting

2025-02-17 · Zihao Wu, Juncheng Dong, Haoming Yang, Vahid Tarokh

Time series forecasting has recently achieved significant progress with multi-scale models to address the heterogeneity between long and short range patterns. Despite their state-of-the-art performance, we identify two p…

Mamba Time Series Time Series Forecasting