A Transformer-based Framework for Multivariate Time Series Representation Learning
In this work we propose for the first time a transformer-based framework for unsupervised representation learning of multivariate time series. Pre-trained models can be potentially used for downstream tasks such as regression and classification, forecasting and missing value imputation. By evaluating our models on several benchmark datasets for multivariate time series regression and classification, we show that not only does our modeling approach represent the most successful method employing unsupervised learning of multivariate time series presented to date, but also that it exceeds the current state-of-the-art performance of supervised methods; it does so even when the number of training samples is very limited, while offering computational efficiency. Finally, we demonstrate that unsupervised pre-training of our transformer models offers a substantial performance benefit over fully supervised learning, even without leveraging additional unlabeled data, i.e., by reusing the same data samples through the unsupervised objective.
Code (6)
Tasks
Computational EfficiencyGeneral ClassificationImputationregressionRepresentation LearningTime SeriesTime Series AnalysisTime Series ClassificationTime Series RegressionUnsupervised Pre-trainingSimilar Papers 제목 키워드 기반
Expressing Multivariate Time Series as Graphs with Time Series Attention Transformer
A reliable and efficient representation of multivariate time series is crucial in various downstream machine learning tasks. In multivariate time series forecasting, each variable depends on its historical values and the…
Multivariate Time Series ForecastingRepresentation LearningTime SeriesTime Series Analysis+1Patch-Level Tokenization with CNN Encoders and Attention for Improved Transformer Time-Series Forecasting
Transformer-based models have shown strong performance in time-series forecasting by leveraging self-attention to model long-range temporal dependencies. However, their effectiveness depends critically on the quality and…
Representation LearningEVOTS: Evolutionary Transformer Search for Time Series Forecasting
Evolutionary neural architecture design for multivariate time-series forecasting remains underexplored, with most approaches relying on fixed Transformer architectures despite substantial variation across tasks and forec…
Neural Architecture SearchTime Series ForecastingHDT: Hierarchical Discrete Transformer for Multivariate Time Series Forecasting
Generative models have gained significant attention in multivariate time series forecasting (MTS), particularly due to their ability to generate high-fidelity samples. Forecasting the probability distribution of multivar…
Multivariate Time Series ForecastingTime SeriesTime Series ForecastingEdgeConvFormer: Dynamic Graph CNN and Transformer based Anomaly Detection in Multivariate Time Series
Transformer-based models for anomaly detection in multivariate time series can benefit from the self-attention mechanism due to its advantage in modeling long-term dependencies. However, Transformer-based anomaly detecti…
Anomaly DetectionTime Series