Motion Sensitive Contrastive Learning for Self-supervised Video Representation
Contrastive learning has shown great potential in video representation learning. However, existing approaches fail to sufficiently exploit short-term motion dynamics, which are crucial to various down-stream video understanding tasks. In this paper, we propose Motion Sensitive Contrastive Learning (MSCL) that injects the motion information captured by optical flows into RGB frames to strengthen feature learning. To achieve this, in addition to clip-level global contrastive learning, we develop Local Motion Contrastive Learning (LMCL) with frame-level contrastive objectives across the two modalities. Moreover, we introduce Flow Rotation Augmentation (FRA) to generate extra motion-shuffled negative samples and Motion Differential Sampling (MDS) to accurately screen training samples. Extensive experiments on standard benchmarks validate the effectiveness of the proposed method. With the commonly-used 3D ResNet-18 as the backbone, we achieve the top-1 accuracies of 91.5\% on UCF101 and 50.3\% on Something-Something v2 for video classification, and a 65.6\% Top-1 Recall on UCF101 for video retrieval, notably improving the state-of-the-art.
Code (0)
등록된 구현이 없습니다.
Tasks
Contrastive LearningRepresentation LearningRetrievalVideo ClassificationVideo RetrievalVideo UnderstandingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Self-Supervised Video Representation Learning with Motion-Contrastive Perception
Visual-only self-supervised learning has achieved significant improvement in video representation learning. Existing related methods encourage models to learn video representations by utilizing contrastive learning or de…
Contrastive LearningRepresentation LearningSelf-Supervised LearningSCVRL: Shuffled Contrastive Video Representation Learning
We propose SCVRL, a novel contrastive-based framework for self-supervised learning for videos. Differently from previous contrast learning based methods that mostly focus on learning visual semantics (e.g., CVRL), SCVRL …
Contrastive LearningRepresentation LearningSelf-Supervised LearningSelf-supervised Video Representation Learning by Pace Prediction
This paper addresses the problem of self-supervised video representation learning from a new perspective -- by video pace prediction. It stems from the observation that human visual system is sensitive to video pace, e.g…
Action RecognitionContrastive LearningPredictionRepresentation Learning+2MaCLR: Motion-aware Contrastive Learning of Representations for Videos
We present MaCLR, a novel method to explicitly perform cross-modal self-supervised video representations learning from visual and motion modalities. Compared to previous video representation learning methods that mostly …
Action DetectionAction RecognitionContrastive LearningRepresentation LearningSLIC: Self-Supervised Learning with Iterative Clustering for Human Action Videos
Self-supervised methods have significantly closed the gap with end-to-end supervised learning for image classification. In the case of human action videos, however, where both appearance and motion are significant factor…
Action ClassificationClusteringContrastive Learningimage-classification+5