Self-supervised Video Representation Learning by Pace Prediction
This paper addresses the problem of self-supervised video representation learning from a new perspective -- by video pace prediction. It stems from the observation that human visual system is sensitive to video pace, e.g., slow motion, a widely used technique in film making. Specifically, given a video played in natural pace, we randomly sample training clips in different paces and ask a neural network to identify the pace for each video clip. The assumption here is that the network can only succeed in such a pace reasoning task when it understands the underlying video content and learns representative spatio-temporal features. In addition, we further introduce contrastive learning to push the model towards discriminating different paces by maximizing the agreement on similar video content. To validate the effectiveness of the proposed method, we conduct extensive experiments on action recognition and video retrieval tasks with several alternative network architectures. Experimental evaluations show that our approach achieves state-of-the-art performance for self-supervised video representation learning across different network architectures and different benchmarks. The code and pre-trained models are available at https://github.com/laura-wang/video-pace.
Code (1)
Tasks
Action RecognitionContrastive LearningPredictionRepresentation LearningRetrievalVideo RetrievalMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
FILS: Self-Supervised Video Feature Prediction In Semantic Language Space
This paper demonstrates a self-supervised approach for learning semantic video representations. Recent vision studies show that a masking strategy for vision and natural language supervision has contributed to developing…
Action RecognitionDecoderHomE: Homography-Equivariant Video Representation Learning
Recent advances in self-supervised representation learning have enabled more efficient and robust model performance without relying on extensive labeled data. However, most works are still focused on images, with few wor…
Action ClassificationAction RecognitionRepresentation LearningSelf-Supervised LearningAuxiliary Learning for Self-Supervised Video Representation via Similarity-based Knowledge Distillation
Despite the outstanding success of self-supervised pretraining methods for video representation learning, they generalise poorly when the unlabeled dataset for pretraining is small or the domain difference between unlabe…
Auxiliary LearningKnowledge DistillationRepresentation LearningMotion Segmentation using Frequency Domain Transformer Networks
Self-supervised prediction is a powerful mechanism to learn representations that capture the underlying structure of the data. Despite recent progress, the self-supervised video prediction task is still challenging. One …
Motion SegmentationPredictionVideo PredictionSelf-Supervised Visual Planning with Temporal Skip Connections
In order to autonomously learn wide repertoires of complex skills, robots must be able to learn from their own autonomously collected data, without human supervision. One learning signal that is always available for auto…
PredictionVideo Prediction