Correlated Attention in Transformers for Multivariate Time Series
Multivariate time series (MTS) analysis prevails in real-world applications such as finance, climate science and healthcare. The various self-attention mechanisms, the backbone of the state-of-the-art Transformer-based models, efficiently discover the temporal dependencies, yet cannot well capture the intricate cross-correlation between different features of MTS data, which inherently stems from complex dynamical systems in practice. To this end, we propose a novel correlated attention mechanism, which not only efficiently captures feature-wise dependencies, but can also be seamlessly integrated within the encoder blocks of existing well-known Transformers to gain efficiency improvement. In particular, correlated attention operates across feature channels to compute cross-covariance matrices between queries and keys with different lag values, and selectively aggregate representations at the sub-series level. This architecture facilitates automated discovery and representation learning of not only instantaneous but also lagged cross-correlations, while inherently capturing time series auto-correlation. When combined with prevalent Transformer baselines, correlated attention mechanism constitutes a better alternative for encoder-only architectures, which are suitable for a wide range of tasks including imputation, anomaly detection and classification. Extensive experiments on the aforementioned tasks consistently underscore the advantages of correlated attention mechanism in enhancing base Transformer models, and demonstrate our state-of-the-art results in imputation, anomaly detection and classification.
Code (0)
등록된 구현이 없습니다.
Tasks
Anomaly DetectionImputationRepresentation LearningTime SeriesMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Using Pre-trained LLMs for Multivariate Time Series Forecasting
Pre-trained Large Language Models (LLMs) encapsulate large amounts of knowledge and take enormous amounts of compute to train. We make use of this resource, together with the observation that LLMs are able to transfer kn…
DecoderMultivariate Time Series ForecastingTime SeriesTime Series ForecastingMICA: Multivariate Infini Compressive Attention for Time Series Forecasting
Multivariate forecasting with Transformers faces a core scalability challenge: modeling cross-channel dependencies via attention compounds attention's quadratic sequence complexity with quadratic channel scaling, making …
Time Series ForecastingSTING: Self-attention based Time-series Imputation Networks using GAN
Time series data are ubiquitous in real-world applications. However, one of the most common problems is that the time series data could have missing values by the inherent nature of the data collection process. So imputi…
ImputationMissing ValuesTime SeriesTime Series AnalysisA Deep Structural Model for Analyzing Correlated Multivariate Time Series
Multivariate time series are routinely encountered in real-world applications, and in many cases, these time series are strongly correlated. In this paper, we present a deep learning structural time series model which ca…
Time SeriesTime Series AnalysisDeep Generative Transformers for Probabilistic Time Series and Spatiotemporal Forecasting
Reliable uncertainty quantification is paramount for forecasting multivariate time series and spatiotemporal data. While Transformer architectures excel at sequence modeling, current probabilistic approaches typically re…