Self-supervised Video Representation Learning with Cross-Stream Prototypical Contrasting
Instance-level contrastive learning techniques, which rely on data augmentation and a contrastive loss function, have found great success in the domain of visual representation learning. They are not suitable for exploiting the rich dynamical structure of video however, as operations are done on many augmented instances. In this paper we propose "Video Cross-Stream Prototypical Contrasting", a novel method which predicts consistent prototype assignments from both RGB and optical flow views, operating on sets of samples. Specifically, we alternate the optimization process; while optimizing one of the streams, all views are mapped to one set of stream prototype vectors. Each of the assignments is predicted with all views except the one matching the prediction, pushing representations closer to their assigned prototypes. As a result, more efficient video embeddings with ingrained motion information are learned, without the explicit need for optical flow computation during inference. We obtain state-of-the-art results on nearest-neighbour video retrieval and action recognition, outperforming previous best by +3.2% on UCF101 using the S3D backbone (90.5% Top-1 acc), and by +7.2% on UCF101 and +15.1% on HMDB51 using the R(2+1)D backbone.
Code (1)
Tasks
Action RecognitionAction Recognition In VideosContrastive LearningData AugmentationOptical Flow EstimationRepresentation LearningRetrievalSelf-Supervised Action RecognitionSelf-Supervised LearningSelf-supervised Video RetrievalVideo ClassificationVideo RecognitionVideo RetrievalMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Vi-MIX FOR SELF-SUPERVISED VIDEO REPRESENTATION
Contrastive representation learning of videos highly rely on exhaustive data aug- mentation strategies. Therefore, towards designing video augmentation for self- supervised learning, we first analyze the best strategy to…
Action RecognitionRepresentation LearningRetrievalSelf-Supervised Learning+1Cross-modal Manifold Cutmix for Self-supervised Video Representation Learning
Contrastive representation learning of videos highly relies on the availability of millions of unlabelled videos. This is practical for videos available on web but acquiring such large scale of videos for real-world appl…
Action RecognitionRepresentation LearningRetrievalSelf-Supervised Learning+1Self-Supervised Learning of Compressed Video Representations
Self-supervised learning of video representations has recently received great attention. Existing methods typically require frames to be decoded before being processed, which increases compute and storage requirements an…
Computational EfficiencySelf-Supervised LearningSelf-Supervised Representation Learning for Visual Anomaly Detection
Self-supervised learning allows for better utilization of unlabelled data. The feature representation obtained by self-supervision can be used in downstream tasks such as classification, object detection, segmentation, a…
Anomaly DetectionGeneral Classificationobject-detectionObject Detection+4When the Future Becomes the Past: Taming Temporal Correspondence for Self-supervised Video Representation Learning
The past decade has witnessed notable achievements in self-supervised learning for video tasks. Recent efforts typically adopt the Masked Video Modeling (MVM) paradigm, leading to significant progress on multiple video t…
Representation LearningSelf-Supervised Learning