Self-Supervised Video Representation Learning with Space-Time Cubic Puzzles
Self-supervised tasks such as colorization, inpainting and zigsaw puzzle have been utilized for visual representation learning for still images, when the number of labeled images is limited or absent at all. Recently, this worthwhile stream of study extends to video domain where the cost of human labeling is even more expensive. However, the most of existing methods are still based on 2D CNN architectures that can not directly capture spatio-temporal information for video applications. In this paper, we introduce a new self-supervised task called as \textit{Space-Time Cubic Puzzles} to train 3D CNNs using large scale video dataset. This task requires a network to arrange permuted 3D spatio-temporal crops. By completing \textit{Space-Time Cubic Puzzles}, the network learns both spatial appearance and temporal relation of video frames, which is our final goal. In experiments, we demonstrate that our learned 3D representation is well transferred to action recognition tasks, and outperforms state-of-the-art 2D CNN-based competitors on UCF101 and HMDB51 datasets.
Code (0)
등록된 구현이 없습니다.
Tasks
Action RecognitionColorizationRepresentation LearningSelf-Supervised Action RecognitionTemporal Action LocalizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Self-supervised and Weakly Supervised Contrastive Learning for Frame-wise Action Representations
Previous work on action representation learning focused on global representations for short video clips. In contrast, many practical applications, such as video alignment, strongly demand learning the intensive represent…
Action ClassificationContrastive LearningRepresentation LearningRetrieval+2Video 3D Sampling for Self-supervised Representation Learning
Most of the existing video self-supervised methods mainly leverage temporal signals of videos, ignoring that the semantics of moving objects and environmental information are all critical for video-related tasks. In this…
Action RecognitionRepresentation LearningRetrievalVideo RetrievalLearning Correspondence from the Cycle-Consistency of Time
We introduce a self-supervised method for learning visual correspondence from unlabeled video. The main idea is to use cycle-consistency in time as free supervisory signal for learning visual representations from scratch…
Optical Flow EstimationSemantic SegmentationSemi-Supervised Video Object SegmentationUnsupervised Video Object Segmentation+2FILS: Self-Supervised Video Feature Prediction In Semantic Language Space
This paper demonstrates a self-supervised approach for learning semantic video representations. Recent vision studies show that a masking strategy for vision and natural language supervision has contributed to developing…
Action RecognitionDecoderModelling Neighbor Relation in Joint Space-Time Graph for Video Correspondence Learning
This paper presents a self-supervised method for learning reliable visual correspondence from unlabeled videos. We formulate the correspondence as finding paths in a joint space-time graph, where nodes are grid patches s…
Contrastive LearningRelation