Contrastive Spatio-Temporal Pretext Learning for Self-supervised Video Representation
Spatio-temporal representation learning is critical for video self-supervised representation. Recent approaches mainly use contrastive learning and pretext tasks. However, these approaches learn representation by discriminating sampled instances via feature similarity in the latent space while ignoring the intermediate state of the learned representations, which limits the overall performance. In this work, taking into account the degree of similarity of sampled instances as the intermediate state, we propose a novel pretext task - spatio-temporal overlap rate (STOR) prediction. It stems from the observation that humans are capable of discriminating the overlap rates of videos in space and time. This task encourages the model to discriminate the STOR of two generated samples to learn the representations. Moreover, we employ a joint optimization combining pretext tasks with contrastive learning to further enhance the spatio-temporal representation learning. We also study the mutual influence of each component in the proposed scheme. Extensive experiments demonstrate that our proposed STOR task can favor both contrastive learning and pretext tasks. The joint optimization scheme can significantly improve the spatio-temporal representation in video understanding. The code is available at https://github.com/Katou2/CSTP.
Code (1)
Tasks
Contrastive LearningRepresentation LearningVideo UnderstandingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Contextualized Spatio-Temporal Contrastive Learning with Self-Supervision
Modern self-supervised learning algorithms typically enforce persistency of instance representations across views. While being very effective on learning holistic image and video representations, such an objective become…
Action LocalizationAction RecognitionContrastive LearningObject Tracking+3Self-Supervised Video Representation Learning with Constrained Spatiotemporal Jigsaw
This paper proposes a novel pretext task for self-supervised video representation learning by exploiting spatiotemporal continuity in videos. It is motivated by the fact that videos are spatiotemporal by nature and a rep…
Representation LearningPretext-Contrastive Learning: Toward Good Practices in Self-supervised Video Representation Leaning
Recently, pretext-task based methods are proposed one after another in self-supervised video feature learning. Meanwhile, contrastive learning methods also yield good performance. Usually, new methods can beat previous o…
Contrastive LearningData AugmentationSelf-Supervised Action RecognitionSelf-Supervised Learning+2Self-Supervised Learning for Group Equivariant Neural Networks
This paper proposes a method to construct pretext tasks for self-supervised learning on group equivariant neural networks. Group equivariant neural networks are the models whose structure is restricted to commute with th…
Self-Supervised LearningBack to the Future: Cycle Encoding Prediction for Self-supervised Contrastive Video Representation Learning
In this paper we show that learning video feature spaces in which temporal cycles are maximally predictable benefits action classification. In particular, we propose a novel learning approach termed Cycle Encoding Predic…
Action ClassificationAction RecognitionRepresentation Learning