Self-Supervised Contrastive Learning for Videos using Differentiable Local Alignment
Robust frame-wise embeddings are essential to perform video analysis and understanding tasks. We present a self-supervised method for representation learning based on aligning temporal video sequences. Our framework uses a transformer-based encoder to extract frame-level features and leverages them to find the optimal alignment path between video sequences. We introduce the novel Local-Alignment Contrastive (LAC) loss, which combines a differentiable local alignment loss to capture local temporal dependencies with a contrastive loss to enhance discriminative learning. Prior works on video alignment have focused on using global temporal ordering across sequence pairs, whereas our loss encourages identifying the best-scoring subsequence alignment. LAC uses the differentiable Smith-Waterman (SW) affine method, which features a flexible parameterization learned through the training phase, enabling the model to adjust the temporal gap penalty length dynamically. Evaluations show that our learned representations outperform existing state-of-the-art approaches on action recognition tasks.
Code (1)
Tasks
Action RecognitionContrastive LearningRepresentation LearningVideo AlignmentSimilar Papers 제목 키워드 기반
Nearest-Neighbor Inter-Intra Contrastive Learning from Unlabeled Videos
Contrastive learning has recently narrowed the gap between self-supervised and supervised methods in image and video domain. State-of-the-art video contrastive learning methods such as CVRL and $\rho$-MoCo spatiotemporal…
Contrastive LearningDiversityTemporal Cycle-Consistency Learning
We introduce a self-supervised representation learning method based on the task of temporal alignment between videos. The method trains a network using temporal cycle consistency (TCC), a differentiable cycle-consistency…
Anomaly DetectionRepresentation LearningSelf-Supervised LearningVideo AlignmentPointCMP: Contrastive Mask Prediction for Self-supervised Learning on Point Cloud Videos
Self-supervised learning can extract representations of good quality from solely unlabeled data, which is appealing for point cloud videos due to their high labelling cost. In this paper, we propose a contrastive mask pr…
Self-Supervised LearningTransfer LearningCHAIN: Exploring Global-Local Spatio-Temporal Information for Improved Self-Supervised Video Hashing
Compressing videos into binary codes can improve retrieval speed and reduce storage overhead. However, learning accurate hash codes for video retrieval can be challenging due to high local redundancy and complex global d…
Contrastive LearningRetrievalVideo RetrievalContrastive Self-Supervised Learning for Spatio-Temporal Analysis of Lung Ultrasound Videos
Self-supervised learning (SSL) methods have shown promise for medical imaging applications by learning meaningful visual representations, even when the amount of labeled data is limited. Here, we extend state-of-the-art …
Contrastive LearningSelf-Supervised Learning