Self-supervised Spatiotemporal Representation Learning by Exploiting Video Continuity
Recent self-supervised video representation learning methods have found significant success by exploring essential properties of videos, e.g. speed, temporal order, etc. This work exploits an essential yet under-explored property of videos, the video continuity, to obtain supervision signals for self-supervised representation learning. Specifically, we formulate three novel continuity-related pretext tasks, i.e. continuity justification, discontinuity localization, and missing section approximation, that jointly supervise a shared backbone for video representation learning. This self-supervision approach, termed as Continuity Perception Network (CPNet), solves the three tasks altogether and encourages the backbone network to learn local and long-ranged motion and context representations. It outperforms prior arts on multiple downstream tasks, such as action recognition, video retrieval, and action localization. Additionally, the video continuity can be complementary to other coarse-grained video properties for representation learning, and integrating the proposed pretext task to prior arts can yield much performance gains.
Code (0)
등록된 구현이 없습니다.
Tasks
Action LocalizationAction RecognitionRepresentation LearningRetrievalVideo RetrievalSimilar Papers 제목 키워드 기반
Self-Supervised Video Representation Learning with Constrained Spatiotemporal Jigsaw
This paper proposes a novel pretext task for self-supervised video representation learning by exploiting spatiotemporal continuity in videos. It is motivated by the fact that videos are spatiotemporal by nature and a rep…
Representation LearningSelf-supervised Representation Learning Framework for Remote Physiological Measurement Using Spatiotemporal Augmentation Loss
Recent advances in supervised deep learning methods are enabling remote measurements of photoplethysmography-based physiological signals using facial videos. The performance of these supervised methods, however, are depe…
Contrastive LearningData AugmentationHeart rate estimationRepresentation Learning+1Self-Supervised Spatiotemporal Learning via Video Clip Order Prediction
We propose a self-supervised spatiotemporal learning technique which leverages the chronological order of videos. Our method can learn the spatiotemporal representation of the video by predicting the order of shuffled cl…
Action RecognitionRetrievalSelf-Supervised Action RecognitionTemporal Action LocalizationSelf-Supervised Spatiotemporal Feature Learning via Video Rotation Prediction
The success of deep neural networks generally requires a vast amount of training data to be labeled, which is expensive and unfeasible in scale, especially for video collections. To alleviate this problem, in this paper,…
Action RecognitionPredictionSelf-Supervised Action RecognitionTemporal Action Localization+1Learning Natural Consistency Representation for Face Forgery Video Detection
Face Forgery videos have elicited critical social public concerns and various detectors have been proposed. However, fully-supervised detectors may lead to easily overfitting to specific forgery methods or videos, and ex…
Representation LearningVideo Classification