Enhancing Masked Time-Series Modeling via Dropping Patches
This paper explores how to enhance existing masked time-series modeling by randomly dropping sub-sequence level patches of time series. On this basis, a simple yet effective method named DropPatch is proposed, which has two remarkable advantages: 1) It improves the pre-training efficiency by a square-level advantage; 2) It provides additional advantages for modeling in scenarios such as in-domain, cross-domain, few-shot learning and cold start. This paper conducts comprehensive experiments to verify the effectiveness of the method and analyze its internal mechanism. Empirically, DropPatch strengthens the attention mechanism, reduces information redundancy and serves as an efficient means of data augmentation. Theoretically, it is proved that DropPatch slows down the rate at which the Transformer representations collapse into the rank-1 linear subspace by randomly dropping patches, thus optimizing the quality of the learned representations
Code (1)
Tasks
Cross-Domain Few-Shotcross-domain few-shot learningData AugmentationFew-Shot LearningTime SeriesMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
SimMTM: A Simple Pre-Training Framework for Masked Time-Series Modeling
Time series analysis is widely used in extensive areas. Recently, to reduce labeling expenses and benefit various tasks, self-supervised pre-training has attracted immense interest. One mainstream paradigm is masked mode…
Representation LearningTime SeriesTime Series AnalysisExplainable Time Series Anomaly Detection using Masked Latent Generative Modeling
We present a novel time series anomaly detection method that achieves excellent detection accuracy while offering a superior level of explainability. Our proposed method, TimeVQVAE-AD, leverages masked generative modelin…
Anomaly DetectionTime SeriesTime Series Anomaly DetectionTime Series GenerationTimeMAE: Self-Supervised Representations of Time Series with Decoupled Masked Autoencoders
Enhancing the expressive capacity of deep learning-based time series models with self-supervised pre-training has become ever-increasingly prevalent in time series classification. Even though numerous efforts have been d…
Time SeriesTime Series AnalysisTime Series ClassificationSelf-Supervised Time Series Representation Learning via Cross Reconstruction Transformer
Unsupervised/self-supervised representation learning in time series is critical since labeled samples are usually scarce in real-world scenarios. Existing approaches mainly leverage the contrastive learning framework, wh…
Contrastive LearningRepresentation LearningTime SeriesTime Series AnalysisLearning to Embed Time Series Patches Independently
Masked time series modeling has recently gained much attention as a self-supervised representation learning strategy for time series. Inspired by masked image modeling in computer vision, recent works first patchify and …
Contrastive LearningRepresentation LearningTime SeriesTime Series Forecasting