Local Attention Mechanism: Boosting the Transformer Architecture for Long-Sequence Time Series Forecasting

2024-10-04 · Ignacio Aguilera-Martos, Andrés Herrera-Poyatos, Julián Luengo, Francisco Herrera

Transformers have become the leading choice in natural language processing over other deep learning architectures. This trend has also permeated the field of time series analysis, especially for long-horizon forecasting, showcasing promising results both in performance and running time. In this paper, we introduce Local Attention Mechanism (LAM), an efficient attention mechanism tailored for time series analysis. This mechanism exploits the continuity properties of time series to reduce the number of attention scores computed. We present an algorithm for implementing LAM in tensor algebra that runs in time and memory O(nlogn), significantly improving upon the O(n^2) time and memory complexity of traditional attention mechanisms. We also note the lack of proper datasets to evaluate long-horizon forecast models. Thus, we propose a novel set of datasets to improve the evaluation of models addressing long-horizon forecasting challenges. Our experimental analysis demonstrates that the vanilla transformer architecture magnified with LAM surpasses state-of-the-art models, including the vanilla attention mechanism. These results confirm the effectiveness of our approach and highlight a range of future challenges in long-sequence time series forecasting.

📄 PDF Abstract BibTeX arXiv:2410.03805

Code (1)

ari-dasci/S-LAM 공식 구현 pytorch

Tasks

tensor algebraTime SeriesTime Series Analysis Time Series Forecasting

Methods 이 논문이 사용한 방법론

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Attention 설명 없음

SET Dynamic Sparse Training method where weight mask is updated randomly periodically

Boosting Crowd Counting via Multifaceted Attention

2022-03-05 · CVPR 2022 1 · Hui Lin, Zhiheng Ma, Rongrong Ji, YaoWei Wang 외

This paper focuses on the challenging crowd counting task. As large-scale variations often exist within crowd images, neither fixed-size convolution kernel of CNN nor fixed-size attention of recent vision transformers ca…

Crowd Counting

Gradient Boosting within a Single Attention Layer

2026-04-03 · Saleh Sargolzaei arxiv

Transformer attention computes a single softmax-weighted average over values -- a one-pass estimate that cannot correct its own errors. We introduce \emph{gradient-boosted attention}, which applies the principle of gradi…

KVT: k-NN Attention for Boosting Vision Transformers

2021-05-28 · Pichao Wang, Xue Wang, Fan Wang, Ming Lin 외

Convolutional Neural Networks (CNNs) have dominated computer vision for years, due to its ability in capturing locality and translation invariance. Recently, many vision transformer architectures have been proposed and t…

PeriodNet: Boosting the Potential of Attention Mechanism for Time Series Forecasting

2025-11-23 · Bowen Zhao, Huanlai Xing, Zhiwen Xiao, Jincheng Peng 외 arxiv

The attention mechanism has demonstrated remarkable potential in sequence modeling, exemplified by its successful application in natural language processing with models such as Bidirectional Encoder Representations from …

Time Series Forecasting

TimeExpert: Boosting Long Time Series Forecasting with Temporal Mix of Experts

2025-09-27 · Xiaowen Ma, Shuning Ge, Fan Yang, Xiangyu Li 외 arxiv

Transformer-based architectures dominate time series modeling by enabling global attention over all timestamps, yet their rigid 'one-size-fits-all' context aggregation fails to address two critical challenges in real-wor…