Revisiting Attention for Multivariate Time Series Forecasting
Current Transformer methods for Multivariate Time-Series Forecasting (MTSF) are all based on the conventional attention mechanism. They involve sequence embedding and performing a linear projection of Q, K, and V, and then computing attention within this latent space. We have never delved into the attention mechanism to explore whether such a mapping space is optimal for MTSF. To investigate this issue, this study first proposes Frequency Spectrum attention (FSatten), a novel attention mechanism based on the frequency domain space. It employs the Fourier transform for embedding and introduces Multi-head Spectrum Scaling (MSS) to replace the conventional linear mapping of Q and K. FSatten can accurately capture the periodic dependencies between sequences and outperform the conventional attention without changing mainstream architectures. We further design a more general method dubbed Scaled Orthogonal attention (SOatten). We propose an orthogonal embedding and a Head-Coupling Convolution (HCC) based on the neighboring similarity bias to guide the model in learning comprehensive dependency patterns. Experiments show that FSatten and SOatten surpass the SOTA which uses conventional attention, making it a good alternative as a basic attention mechanism for MTSF. The codes and log files will be released at: https://github.com/Joeland4/FSatten-SOatten.
Code (1)
Tasks
Multivariate Time Series ForecastingTime SeriesTime Series ForecastingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
TwinS: Revisiting Non-Stationarity in Multivariate Time Series Forecasting
Recently, multivariate time series forecasting tasks have garnered increasing attention due to their significant practical applications, leading to the emergence of various deep forecasting models. However, real-world ti…
Multivariate Time Series ForecastingTime SeriesTime Series ForecastingThe Capacity and Robustness Trade-off: Revisiting the Channel Independent Strategy for Multivariate Time Series Forecasting
Multivariate time series data comprises various channels of variables. The multivariate forecasting models need to capture the relationship between the channels to accurately predict future values. However, recently, the…
Multivariate Time Series ForecastingTime SeriesTime Series ForecastingRevisiting Multivariate Time Series Forecasting with Missing Values
Missing values are common in real-world time series, and multivariate time series forecasting with missing values (MTSF-M) has become a crucial area of research for ensuring reliable predictions. To address the challenge…
Multivariate Time Series ForecastingExpressing Multivariate Time Series as Graphs with Time Series Attention Transformer
A reliable and efficient representation of multivariate time series is crucial in various downstream machine learning tasks. In multivariate time series forecasting, each variable depends on its historical values and the…
Multivariate Time Series ForecastingRepresentation LearningTime SeriesTime Series Analysis+1DSANet: Dual Self-Attention Network for Multivariate Time Series Forecasting
Multivariate time series forecasting has attracted wide attention in areas, such as system, traffic, and finance. The difficulty of the task lies in that traditional methods fail to capture complicated nonlinear dependen…
Multivariate Time Series ForecastingTime SeriesTime Series AnalysisTime Series Forecasting