Cross-Architecture Self-supervised Video Representation Learning
In this paper, we present a new cross-architecture contrastive learning (CACL) framework for self-supervised video representation learning. CACL consists of a 3D CNN and a video transformer which are used in parallel to generate diverse positive pairs for contrastive learning. This allows the model to learn strong representations from such diverse yet meaningful pairs. Furthermore, we introduce a temporal self-supervised learning module able to predict an Edit distance explicitly between two video sequences in the temporal order. This enables the model to learn a rich temporal representation that compensates strongly to the video-level representation learned by the CACL. We evaluate our method on the tasks of video retrieval and action recognition on UCF101 and HMDB51 datasets, where our method achieves excellent performance, surpassing the state-of-the-art methods such as VideoMoCo and MoCo+BE by a large margin. The code is made available at https://github.com/guoshengcv/CACL.
Code (1)
Tasks
Action RecognitionContrastive LearningRepresentation LearningRetrievalSelf-Supervised LearningVideo RetrievalMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Cycle-Contrast for Self-Supervised Video Representation Learning
We present Cycle-Contrastive Learning (CCL), a novel self-supervised method for learning video representation. Following a nature that there is a belong and inclusion relation of video and its frames, CCL is designed to …
Action RecognitionContrastive LearningRepresentation LearningRetrieval+1A Large-Scale Analysis on Self-Supervised Video Representation Learning
Self-supervised learning is an effective way for label-free model pre-training, especially in the video domain where labeling is expensive. Existing self-supervised works in the video domain use varying experimental setu…
BenchmarkingRepresentation LearningSelf-Supervised Action RecognitionSelf-Supervised LearningSelf-Supervised Learning of Compressed Video Representations
Self-supervised learning of video representations has recently received great attention. Existing methods typically require frames to be decoded before being processed, which increases compute and storage requirements an…
Computational EfficiencySelf-Supervised LearningSelf-supervised Video Representation Learning by Pace Prediction
This paper addresses the problem of self-supervised video representation learning from a new perspective -- by video pace prediction. It stems from the observation that human visual system is sensitive to video pace, e.g…
Action RecognitionContrastive LearningPredictionRepresentation Learning+2Multi-entity Video Transformers for Fine-Grained Video Representation Learning
The area of temporally fine-grained video representation learning aims to generate frame-by-frame representations for temporally dense tasks. In this work, we advance the state-of-the-art for this area by re-examining th…
Representation Learning