Long-term series forecasting with Query Selector -- efficient model of sparse attention

2021-07-19 · Jacek Klimek, Jakub Klimek, Witold Kraskiewicz, Mateusz Topolewski

Various modifications of TRANSFORMER were recently used to solve time-series forecasting problem. We propose Query Selector - an efficient, deterministic algorithm for sparse attention matrix. Experiments show it achieves state-of-the art results on ETT, Helpdesk and BPI'12 datasets.

📄 PDF Abstract BibTeX arXiv:2107.08687

Code (2)

moraieu/query-selector 공식 구현 pytorch

moraieu/query-selector-classification pytorch

Tasks

Multivariate Time Series ForecastingTime SeriesTime Series Analysis Time Series Forecasting

Methods 이 논문이 사용한 방법론

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

Position-Wise Feed-Forward Layer 설명 없음

Residual Connection 설명 없음

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Adam 설명 없음

Preformer: Predictive Transformer with Multi-Scale Segment-wise Correlations for Long-Term Time Series Forecasting

2022-02-23 · Dazhao Du, Bing Su, Zhewei Wei

Transformer-based methods have shown great potential in long-term time series forecasting. However, most of these methods adopt the standard point-wise self-attention mechanism, which not only becomes intractable for lon…

Time Series Time Series Analysis Time Series Forecasting

TimeExpert: Boosting Long Time Series Forecasting with Temporal Mix of Experts

2025-09-27 · Xiaowen Ma, Shuning Ge, Fan Yang, Xiangyu Li 외 arxiv

Transformer-based architectures dominate time series modeling by enabling global attention over all timestamps, yet their rigid 'one-size-fits-all' context aggregation fails to address two critical challenges in real-wor…

Time Series Forecasting

PENGUIN: Enhancing Transformer with Periodic-Nested Group Attention for Long-term Time Series Forecasting

2025-08-19 · Tian Sun, Yuqi Chen, Weiwei Sun arxiv

Despite advances in the Transformer architecture, their effectiveness for long-term time series forecasting (LTSF) remains controversial. In this paper, we investigate the potential of integrating explicit periodicity mo…

Time Series Forecasting

MPPN: Multi-Resolution Periodic Pattern Network For Long-Term Time Series Forecasting

2023-06-12 · Xing Wang, Zhendong Wang, Kexin Yang, Junlan Feng 외

Long-term time series forecasting plays an important role in various real-world scenarios. Recent deep learning methods for long-term series forecasting tend to capture the intricate patterns of time series by decomposit…

Time Series Time Series Forecasting

Adaptive Automatic Model Selection for Demand Forecasting under Heterogeneous Demand Patterns

2026-02-15 · Adolfo González, Víctor Parada arxiv

Demand forecasting is critical for inventory planning, procurement, replenishment, production, and capacity decisions in heterogeneous supply chains. However, selecting the most appropriate model for each demand series r…