TimesURL: Self-supervised Contrastive Learning for Universal Time Series Representation Learning
Learning universal time series representations applicable to various types of downstream tasks is challenging but valuable in real applications. Recently, researchers have attempted to leverage the success of self-supervised contrastive learning (SSCL) in Computer Vision(CV) and Natural Language Processing(NLP) to tackle time series representation. Nevertheless, due to the special temporal characteristics, relying solely on empirical guidance from other domains may be ineffective for time series and difficult to adapt to multiple downstream tasks. To this end, we review three parts involved in SSCL including 1) designing augmentation methods for positive pairs, 2) constructing (hard) negative pairs, and 3) designing SSCL loss. For 1) and 2), we find that unsuitable positive and negative pair construction may introduce inappropriate inductive biases, which neither preserve temporal properties nor provide sufficient discriminative features. For 3), just exploring segment- or instance-level semantics information is not enough for learning universal representation. To remedy the above issues, we propose a novel self-supervised framework named TimesURL. Specifically, we first introduce a frequency-temporal-based augmentation to keep the temporal property unchanged. And then, we construct double Universums as a special kind of hard negative to guide better contrastive learning. Additionally, we introduce time reconstruction as a joint optimization objective with contrastive learning to capture both segment-level and instance-level information. As a result, TimesURL can learn high-quality universal representations and achieve state-of-the-art performance in 6 different downstream tasks, including short- and long-term forecasting, imputation, classification, anomaly detection and transfer learning.
Code (2)
Tasks
Anomaly DetectionContrastive LearningImputationRepresentation LearningTime SeriesTransfer LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Self-Supervised Learning from Contrastive Mixtures for Personalized Speech Enhancement
This work explores how self-supervised learning can be universally used to discover speaker-specific features towards enabling personalized speech enhancement models. We specifically address the few-shot learning scenari…
Contrastive LearningFew-Shot LearningModel CompressionSelf-Supervised Learning+1reCSE: Portable Reshaping Features for Sentence Embedding in Self-supervised Contrastive Learning
We propose reCSE, a self supervised contrastive learning sentence representation framework based on feature reshaping. This framework is different from the current advanced models that use discrete data augmentation meth…
Contrastive LearningData AugmentationGPUSemantic Similarity+4Time-Series Contrastive Learning against False Negatives and Class Imbalance
As an exemplary self-supervised approach for representation learning, time-series contrastive learning has exhibited remarkable advancements in contemporary research. While recent contrastive learning strategies have foc…
Contrastive LearningRepresentation LearningTime SeriesUnifying Graph Contrastive Learning via Graph Message Augmentation
Graph contrastive learning is usually performed by first conducting Graph Data Augmentation (GDA) and then employing a contrastive learning pipeline to train GNNs. As we know that GDA is an important issue for graph cont…
Contrastive LearningData AugmentationGraph LearningSelf-Supervised LearningLearning Universal User Representations via Self-Supervised Lifelong Behaviors Modeling
Universal user representation is an important research topic in industry, and is widely used in diverse downstream user analysis tasks, such as user profiling and user preference prediction. With the rapid development of…
Contrastive LearningDimensionality ReductionRepresentation LearningSelf-Supervised Learning