Representation Learning via Global Temporal Alignment and Cycle-Consistency
We introduce a weakly supervised method for representation learning based on aligning temporal sequences (e.g., videos) of the same process (e.g., human action). The main idea is to use the global temporal ordering of latent correspondences across sequence pairs as a supervisory signal. In particular, we propose a loss based on scoring the optimal sequence alignment to train an embedding network. Our loss is based on a novel probabilistic path finding view of dynamic time warping (DTW) that contains the following three key features: (i) the local path routing decisions are contrastive and differentiable, (ii) pairwise distances are cast as probabilities that are contrastive as well, and (iii) our formulation naturally admits a global cycle consistency loss that verifies correspondences. For evaluation, we consider the tasks of fine-grained action classification, few shot learning, and video synchronization. We report significant performance increases over previous methods. In addition, we report two applications of our temporal alignment framework, namely 3D pose reconstruction and fine-grained audio/visual retrieval.
Code (1)
Tasks
Action ClassificationDynamic Time WarpingFew-Shot LearningRepresentation LearningRetrievalTemporal SequencesVideo SynchronizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Temporal Cycle-Consistency Learning
We introduce a self-supervised representation learning method based on the task of temporal alignment between videos. The method trains a network using temporal cycle consistency (TCC), a differentiable cycle-consistency…
Anomaly DetectionRepresentation LearningSelf-Supervised LearningVideo AlignmentLearning to Predict Activity Progress by Self-Supervised Video Alignment
In this paper we tackle the problem of self-supervised video alignment and activity progress prediction using in-the-wild videos. Our proposed self-supervised representation learning method carefully addresses differ…
Representation LearningVideo AlignmentCycle Label-Consistent Networks for Unsupervised Domain Adaptation
Domain adaptation aims to leverage a labeled source domain to learn a classifier for the unlabeled target domain with a different distribution. Previous methods mostly match the distribution between two domains by global…
Domain AdaptationUnsupervised Domain AdaptationAdaptive Oscillatory-State Alignment for Time Series Forecasting
Long-term time series forecasting benefits from inductive biases that expose recurring temporal structure. Existing periodic forecasting methods typically model recurrence through predefined periods, global spectral comp…
Time Series ForecastingSpatiotemporal Facial Action Unit Detection using Twin Cycle Autoencoders for Driver Monitoring
Driver monitoring systems (DMS) increasingly rely on facial cues to infer drowsiness, distraction, and cognitive load in real time. Facial Action Units (AUs), grounded in the Facial Action Coding System (FACS), provide a…
Facial Action Unit Detection