Self-Supervised Video Representation Learning via Latent Time Navigation
Self-supervised video representation learning aimed at maximizing similarity between different temporal segments of one video, in order to enforce feature persistence over time. This leads to loss of pertinent information related to temporal relationships, rendering actions such as enter' and leave' to be indistinguishable. To mitigate this limitation, we propose Latent Time Navigation (LTN), a time-parameterized contrastive learning strategy that is streamlined to capture fine-grained motions. Specifically, we maximize the representation similarity between different video segments from one video, while maintaining their representations time-aware along a subspace of the latent representation code including an orthogonal basis to represent temporal changes. Our extensive experimental analysis suggests that learning video representations by LTN consistently improves performance of action classification in fine-grained and human-oriented tasks (e.g., on Toyota Smarthome dataset). In addition, we demonstrate that our proposed model, when pre-trained on Kinetics-400, generalizes well onto the unseen real world video benchmark datasets UCF101 and HMDB51, achieving state-of-the-art performance in action recognition.
Code (0)
등록된 구현이 없습니다.
Tasks
Action ClassificationAction RecognitionContrastive LearningRepresentation LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
ViewCLR: Learning Self-supervised Video Representation for Unseen Viewpoints
Learning self-supervised video representation predominantly focuses on discriminating instances generated from simple data augmentation schemes. However, the learned representation often fails to generalize over unseen c…
Data AugmentationSelf-Supervised Learning of Deviation in Latent Representation for Co-speech Gesture Video Generation
Gestures are pivotal in enhancing co-speech communication. While recent works have mostly focused on point-level motion transformation or fully supervised motion representations through data-driven approaches, we explore…
Self-Supervised LearningSSIMVideo GenerationMidway Network: Learning Representations for Recognition and Motion from Latent Dynamics
Object recognition and motion understanding are key components of perception that complement each other. While self-supervised learning methods have shown promise in their ability to learn from unlabeled data, they have …
Self-Supervised LearningSemantic SegmentationObject RecognitionDecision MakingSuppressing Static Visual Cues via Normalizing Flows for Self-Supervised Video Representation Learning
Despite the great progress in video understanding made by deep convolutional neural networks, feature representation learned by existing methods may be biased to static visual cues. To address this issue, we propose a no…
Contrastive LearningRepresentation LearningVideo UnderstandingMomentum-Guided Semantic Forecasting (MoFore) for Self-Supervised Video Representation Learning
Self-supervised video representation learning has recently advanced through contrastive learning, masked reconstruction, and predictive representation learning. Reconstruction-based approaches such as MAE and VideoMAE le…
Representation LearningContrastive Learning