SeCo: Exploring Sequence Supervision for Unsupervised Representation Learning
A steady momentum of innovations and breakthroughs has convincingly pushed the limits of unsupervised image representation learning. Compared to static 2D images, video has one more dimension (time). The inherent supervision existing in such sequential structure offers a fertile ground for building unsupervised learning models. In this paper, we compose a trilogy of exploring the basic and generic supervision in the sequence from spatial, spatiotemporal and sequential perspectives. We materialize the supervisory signals through determining whether a pair of samples is from one frame or from one video, and whether a triplet of samples is in the correct temporal order. We uniquely regard the signals as the foundation in contrastive learning and derive a particular form named Sequence Contrastive Learning (SeCo). SeCo shows superior results under the linear protocol on action recognition (Kinetics), untrimmed activity recognition (ActivityNet) and object tracking (OTB-100). More remarkably, SeCo demonstrates considerable improvements over recent unsupervised pre-training techniques, and leads the accuracy by 2.96% and 6.47% against fully-supervised ImageNet pre-training in action recognition task on UCF101 and HMDB51, respectively. Source code is available at \url{https://github.com/YihengZhang-CV/SeCo-Sequence-Contrastive-Learning}.
Code (3)
Tasks
Action RecognitionActivity RecognitionContrastive LearningObject TrackingRepresentation LearningTripletUnsupervised Pre-trainingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences
In the field of artificial intelligence, a combination of scale in data and model capacity enabled by unsupervised learning has led to major advances in representation learning and statistical generation. In the life sci…
DiversityLanguage ModelingLanguage ModellingRepresentation LearningExploring single-song autoencoding schemes for audio-based music structure analysis
The ability of deep neural networks to learn complex data relations and representations is established nowadays, but it generally relies on large sets of training data. This work explores a "piece-specific" autoencoding …
Speech-to-speech Translation between Untranscribed Unknown Languages
In this paper, we explore a method for training speech-to-speech translation tasks without any transcription or linguistic supervision. Our proposed method consists of two steps: First, we train and generate discrete rep…
Speech-to-Speech TranslationTranslationExploring Author Context for Detecting Intended vs Perceived Sarcasm
We investigate the impact of using author context on textual sarcasm detection. We define author context as the embedded representation of their historical posts on Twitter and suggest neural models that extract these re…
Sarcasm DetectionTAGCross-lingual Supervision Improves Unsupervised Neural Machine Translation
Neural machine translation~(NMT) is ineffective for zero-resource languages. Recent works exploring the possibility of unsupervised neural machine translation (UNMT) with only monolingual data can achieve promising resul…
Machine TranslationNMTTranslation