Self-supervised learning of class embeddings from video
This work explores how to use self-supervised learning on videos to learn a class-specific image embedding that encodes pose and shape information. At train time, two frames of the same video of an object class (e.g. human upper body) are extracted and each encoded to an embedding. Conditioned on these embeddings, the decoder network is tasked to transform one frame into another. To successfully perform long range transformations (e.g. a wrist lowered in one image should be mapped to the same wrist raised in another), we introduce a hierarchical probabilistic network decoder model. Once trained, the embedding can be used for a variety of downstream tasks and domains. We demonstrate our approach quantitatively on three distinct deformable object classes -- human full bodies, upper bodies, faces -- and show experimentally that the learned embeddings do indeed generalise. They achieve state-of-the-art performance in comparison to other self-supervised methods trained on the same datasets, and approach the performance of fully supervised methods.
Code (0)
등록된 구현이 없습니다.
Tasks
DecoderSelf-Supervised LearningSimilar Papers 제목 키워드 기반
Temporal Cycle-Consistency Learning
We introduce a self-supervised representation learning method based on the task of temporal alignment between videos. The method trains a network using temporal cycle consistency (TCC), a differentiable cycle-consistency…
Anomaly DetectionRepresentation LearningSelf-Supervised LearningVideo AlignmentSelf-supervised Learning of Pose Embeddings from Spatiotemporal Relations in Videos
Human pose analysis is presently dominated by deep convolutional networks trained with extensive manual annotations of joint locations and beyond. To avoid the need for expensive labeling, we exploit spatiotemporal relat…
Pose EstimationRetrievalSelf-Supervised LearningTemporally Coherent Embeddings for Self-Supervised Video Representation Learning
This paper presents TCE: Temporally Coherent Embeddings for self-supervised video representation learning. The proposed method exploits inherent structure of unlabeled video data to explicitly enforce temporal coherency …
Action RecognitionMetric LearningRepresentation LearningSelf-Supervised Action Recognition+2Exploring Temporal Granularity in Self-Supervised Video Representation Learning
This work presents a self-supervised learning framework named TeG to explore Temporal Granularity in learning video representations. In TeG, we sample a long clip from a video and a short clip that lies inside the long c…
Representation LearningSelf-Supervised LearningFine-grained Multi-Modal Self-Supervised Learning
Multi-Modal Self-Supervised Learning from videos has been shown to improve model's performance on various downstream tasks. However, such Self-Supervised pre-training requires large batch sizes and a large amount of comp…
Action RecognitionSelf-Supervised Learning