Self-supervised Learning of Interpretable Keypoints from Unlabelled Videos
We propose KeypointGAN, a new method for recognizing the pose of objects from a single image that for learning uses only unlabelled videos and a weak empirical prior on the object poses. Video frames differ primarily in the pose of the objects they contain, so our method distils the pose information by analyzing the differences between frames. The distillation uses a new dual representation of the geometry of objects as a set of 2D keypoints, and as a pictorial representation, i.e. a skeleton image. This has three benefits: (1) it provides a tight `geometric bottleneck' which disentangles pose from appearance, (2) it can leverage powerful image-to-image translation networks to map between photometry and geometry, and (3) it allows to incorporate empirical pose priors in the learning process. The pose priors are obtained from unpaired data, such as from a different dataset or modality such as mocap, such that no annotated image is ever used in learning the pose recognition network. In standard benchmarks for pose recognition for humans and faces, our method achieves state-of-the-art performance among methods that do not require any labelled images for training.
Code (0)
등록된 구현이 없습니다.
Tasks
Facial Landmark DetectionImage-to-Image TranslationKeypoint DetectionSelf-Supervised LearningTranslationUnsupervised Landmark DetectionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Self-Supervised Keypoint Discovery in Behavioral Videos
We propose a method for learning the posture and structure of agents from unlabelled behavioral videos. Starting from the observation that behaving agents are generally the main sources of movement in behavioral videos, …
DecoderUnsupervised Human Pose EstimationSelf-Supervised 3D Keypoint Learning for Ego-motion Estimation
Detecting and matching robust viewpoint-invariant keypoints is critical for visual SLAM and Structure-from-Motion. State-of-the-art learning-based methods generate training samples via homography adaptation to create 2D …
Autonomous VehiclesDepth EstimationGeometric MatchingKeypoint Detection+4Learning Interpretable Low-dimensional Representation via Physical Symmetry
We have recently seen great progress in learning interpretable music representations, ranging from basic factors, such as pitch and timbre, to high-level concepts, such as chord and texture. However, most methods rely he…
counterfactualTime SeriesAutoLink: Self-supervised Learning of Human Skeletons and Object Outlines by Linking Keypoints
Structured representations such as keypoints are widely used in pose transfer, conditional image generation, animation, and 3D reconstruction. However, their supervised learning requires expensive annotation for each tar…
Pose EstimationSelf-Supervised LearningUnsupervised Facial Landmark DetectionUnsupervised Facial Landmark Detection on MAFL+4A Horse with no Labels: Self-Supervised Horse Pose Estimation from Unlabelled Images and Synthetic Prior
Obtaining labelled data to train deep learning methods for estimating animal pose is challenging. Recently, synthetic data has been widely used for pose estimation tasks, but most methods still rely on supervised learnin…
Pose Estimation