Self-supervised Temporal Discriminative Learning for Video Representation Learning
Temporal cues in videos provide important information for recognizing actions accurately. However, temporal-discriminative features can hardly be extracted without using an annotated large-scale video action dataset for training. This paper proposes a novel Video-based Temporal-Discriminative Learning (VTDL) framework in self-supervised manner. Without labelled data for network pretraining, temporal triplet is generated for each anchor video by using segment of the same or different time interval so as to enhance the capacity for temporal feature representation. Measuring temporal information by time derivative, Temporal Consistent Augmentation (TCA) is designed to ensure that the time derivative (in any order) of the augmented positive is invariant except for a scaling constant. Finally, temporal-discriminative features are learnt by minimizing the distance between each anchor and its augmented positive, while the distance between each anchor and its augmented negative as well as other videos saved in the memory bank is maximized to enrich the representation diversity. In the downstream action recognition task, the proposed method significantly outperforms existing related works. Surprisingly, the proposed self-supervised approach is better than fully-supervised methods on UCF101 and HMDB51 when a small-scale video dataset (with only thousands of videos) is used for pre-training. The code has been made publicly available on https://github.com/FingerRec/Self-Supervised-Temporal-Discriminative-Representation-Learning-for-Video-Action-Recognition.
Code (1)
Tasks
Action RecognitionRepresentation LearningTemporal Action LocalizationTripletSimilar Papers 제목 키워드 기반
Video Playback Rate Perception for Self-supervisedSpatio-Temporal Representation Learning
In self-supervised spatio-temporal representation learning, the temporal resolution and long-short term characteristics are not yet fully explored, which limits representation capabilities of learned models. In this pape…
Action RecognitionDecoderRepresentation LearningRetrieval+1Video Playback Rate Perception for Self-Supervised Spatio-Temporal Representation Learning
In self-supervised spatio-temporal representation learning, the temporal resolution and long-short term characteristics are not yet fully explored, which limits representation capabilities of learned models. In this pape…
Action RecognitionDecoderRepresentation LearningRetrieval+3Self-supervised Temporal Learning
Self-supervised learning (SSL) has shown its powerful ability in discriminative representations for various visual, audio, and video applications. However, most recent works still focus on the different paradigms of spat…
Contrastive LearningRetrievalSelf-Supervised LearningVideo Classification+1Spatial-then-Temporal Self-Supervised Learning for Video Correspondence
In low-level video analyses, effective representations are important to derive the correspondences between video frames. These representations have been learned in a self-supervised fashion from unlabeled images or video…
Contrastive LearningSelf-Supervised LearningSpatio-temporal Contrastive Domain Adaptation for Action Recognition
Unsupervised domain adaptation (UDA) for human action recognition is a practical and challenging problem. Compared with image-based UDA, video-based UDA is comprehensive to bridge the domain shift on both spatial rep…
Action RecognitionContrastive LearningDomain AdaptationSelf-Supervised Learning+2