ASCNet: Self-supervised Video Representation Learning with Appearance-Speed Consistency
We study self-supervised video representation learning, which is a challenging task due to 1) lack of labels for explicit supervision; 2) unstructured and noisy visual information. Existing methods mainly use contrastive loss with video clips as the instances and learn visual representation by discriminating instances from each other, but they need a careful treatment of negative pairs by either relying on large batch sizes, memory banks, extra modalities or customized mining strategies, which inevitably includes noisy data. In this paper, we observe that the consistency between positive samples is the key to learn robust video representation. Specifically, we propose two tasks to learn the appearance and speed consistency, respectively. The appearance consistency task aims to maximize the similarity between two clips of the same video with different playback speeds. The speed consistency task aims to maximize the similarity between two clips with the same playback speed but different appearance information. We show that optimizing the two tasks jointly consistently improves the performance on downstream tasks, e.g., action recognition and video retrieval. Remarkably, for action recognition on the UCF-101 dataset, we achieve 90.8\% accuracy without using any extra modalities or negative pairs for unsupervised pretraining, which outperforms the ImageNet supervised pretrained model. Codes and models will be available.
Code (0)
등록된 구현이 없습니다.
Tasks
Action RecognitionRepresentation LearningRetrievalVideo RetrievalSimilar Papers 제목 키워드 기반
Self-Supervised Video GANs: Learning for Appearance Consistency and Motion Coherency
A video can be represented by the composition of appearance and motion. Appearance (or content) expresses the information invariant throughout time, and motion describes the time-variant movement. Here, we propose se…
Contrastive LearningASCNet: Asymmetric Sampling Correction Network for Infrared Image Destriping
In a real-world infrared imaging system, effectively learning a consistent stripe noise removal model is essential. Most existing destriping methods cannot precisely reconstruct images due to cross-level semantic gaps an…
Feature UpsamplingImage ReconstructionMultiview Pseudo-Labeling for Semi-supervised Learning from Video
We present a multiview pseudo-labeling approach to video learning, a novel framework that uses complementary views in the form of appearance and motion information for semi-supervised learning in video. The complementary…
Representation LearningVideo RecognitionContrast and Order Representations for Video Self-Supervised Learning
This paper studies the problem of learning self-supervised representations on videos. In contrast to image modality that only requires appearance information on objects or scenes, video needs to further explore the r…
Action RecognitionSelf-Supervised Action Recognition LinearSelf-Supervised LearningSelf-supervised Video Object Segmentation
The objective of this paper is self-supervised representation learning, with the goal of solving semi-supervised video object segmentation (a.k.a. dense tracking). We make the following contributions: (i) we propose to i…
ObjectOne-shot visual object segmentationRepresentation LearningSegmentation+4