Fourier-Mixed Window Attention: Accelerating Informer for Long Sequence Time-Series Forecasting

2023-07-02 · Nhat Thanh Tran, Jack Xin

We study a fast local-global window-based attention method to accelerate Informer for long sequence time-series forecasting. While window attention being local is a considerable computational saving, it lacks the ability to capture global token information which is compensated by a subsequent Fourier transform block. Our method, named FWin, does not rely on query sparsity hypothesis and an empirical approximation underlying the ProbSparse attention of Informer. Through experiments on univariate and multivariate datasets, we show that FWin transformers improve the overall prediction accuracies of Informer while accelerating its inference speeds by 1.6 to 2 times. We also provide a mathematical definition of FWin attention, and prove that it is equivalent to the canonical full attention under the block diagonal invertibility (BDI) condition of the attention matrix. The BDI is shown experimentally to hold with high probability for typical benchmark datasets.

📄 PDF Abstract BibTeX arXiv:2307.00493

Code (1)

nhatthanhtran/fwin2023 공식 구현 pytorch

Tasks

Time SeriesTime Series Forecasting

Methods 이 논문이 사용한 방법론

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

TKwinFormer: Top k Window Attention in Vision Transformers for Feature Matching

2023-08-29 · Yun Liao, Yide Di, Hao Zhou, Kaijun Zhu 외

Local feature matching remains a challenging task, primarily due to difficulties in matching sparse keypoints and low-texture regions. The key to solving this problem lies in effectively and accurately integrating global…

TempTPI: Informer-Based trajectory prediction for maritime vessels

2026-09-09 · Kevin Ferneding, Veronika Lietavcova, Aleksandra M. Blachowiak, Peder Heiselberg arxiv

Accurate long-term trajectory prediction for maritime vessels is essential for safety and logistical efficiency. While deep learning models, particularly Transformers, have shown promise in processing Automatic Identific…

Trajectory Prediction

Medical Image Segmentation Using Directional Window Attention

2024-06-25 · Daniya Najiha Abdul Kareem, Mustansar Fiaz, Noa Novershtern, Hisham Cholakkal

Accurate segmentation of medical images is crucial for diagnostic purposes, including cell segmentation, tumor identification, and organ localization. Traditional convolutional neural network (CNN)-based approaches strug…

Cell Segmentation Diagnostic Image Segmentation Medical Image Segmentation+3

Sketching as a Tool for Understanding and Accelerating Self-attention for Long Sequences

2021-12-10 · NAACL 2022 7 · Yifan Chen, Qi Zeng, Dilek Hakkani-Tur, Di Jin 외

Transformer-based models are not efficient in processing long sequences due to the quadratic space and time complexity of the self-attention modules. To address this limitation, Linformer and Informer are proposed to red…

DwinFormer: Dual Window Transformers for End-to-End Monocular Depth Estimation

2023-03-06 · Md Awsafur Rahman, Shaikh Anowarul Fattah

Depth estimation from a single image is of paramount importance in the realm of computer vision, with a multitude of applications. Conventional methods suffer from the trade-off between consistency and fine-grained detai…

Decoder Depth Estimation Monocular Depth Estimation