Rethinking Self-supervised Correspondence Learning: A Video Frame-level Similarity Perspective
Learning a good representation for space-time correspondence is the key for various computer vision tasks, including tracking object bounding boxes and performing video object pixel segmentation. To learn generalizable representation for correspondence in large-scale, a variety of self-supervised pretext tasks are proposed to explicitly perform object-level or patch-level similarity learning. Instead of following the previous literature, we propose to learn correspondence using Video Frame-level Similarity (VFS) learning, i.e, simply learning from comparing video frames. Our work is inspired by the recent success in image-level contrastive learning and similarity learning for visual recognition. Our hypothesis is that if the representation is good for recognition, it requires the convolutional features to find correspondence between similar objects or parts. Our experiments show surprising results that VFS surpasses state-of-the-art self-supervised approaches for both OTB visual object tracking and DAVIS video object segmentation. We perform detailed analysis on what matters in VFS and reveals new properties on image and frame level similarity learning. Project page with code is available at https://jerryxu.net/VFS
Code (5)
Tasks
Contrastive LearningObjectObject TrackingSegmentationSemantic SegmentationVideo Object SegmentationVideo Semantic SegmentationVisual Object TrackingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Rethinking Temporal Consistency in Video Object-Centric Learning: From Prediction to Correspondence
The de facto approach in video object-centric learning maintains temporal consistency through learned dynamics modules that predict future object representations, called slots. We demonstrate that these predictors functi…
Leveraging Motion Information for Better Self-Supervised Video Correspondence Learning
Self-supervised video correspondence learning depends on the ability to accurately associate pixels between video frames that correspond to the same visual object. However, achieving reliable pixel matching without super…
ObjectSemantic SegmentationVideo Object SegmentationVideo Semantic SegmentationContrastive Transformation for Self-supervised Correspondence Learning
In this paper, we focus on the self-supervised learning of visual correspondence using unlabeled videos in the wild. Our method simultaneously considers intra- and inter-video representation associations for reliable cor…
Object TrackingSelf-Supervised LearningSemantic SegmentationVideo Object Segmentation+2Locality-Aware Inter-and Intra-Video Reconstruction for Self-Supervised Correspondence Learning
Our target is to learn visual correspondence from unlabeled videos. We develop LIIR, a locality-aware inter-and intra-video reconstruction framework that fills in three missing pieces, i.e., instance discrimination, loca…
PositionRepresentation LearningVideo ReconstructionSelf-supervised Learning for Video Correspondence Flow
The objective of this paper is self-supervised learning of feature embeddings that are suitable for matching correspondences along the videos, which we term correspondence flow. By leveraging the natural spatial-temporal…
Self-Supervised LearningSemi-Supervised Video Object SegmentationUnsupervised Video Object SegmentationVideo Correspondence Flow+2