Self-supervised 3D Shape and Viewpoint Estimation from Single Images for Robotics
We present a convolutional neural network for joint 3D shape prediction and viewpoint estimation from a single input image. During training, our network gets the learning signal from a silhouette of an object in the input image - a form of self-supervision. It does not require ground truth data for 3D shapes and the viewpoints. Because it relies on such a weak form of supervision, our approach can easily be applied to real-world data. We demonstrate that our method produces reasonable qualitative and quantitative results on natural images for both shape estimation and viewpoint prediction. Unlike previous approaches, our method does not require multiple views of the same object instance in the dataset, which significantly expands the applicability in practical robotics scenarios. We showcase it by using the hallucinated shapes to improve the performance on the task of grasping real-world objects both in simulation and with a PR2 robot.
Code (1)
Tasks
3D Object Reconstruction From A Single Image3D Reconstruction3D Shape GenerationRobotic GraspingViewpoint EstimationSimilar Papers 제목 키워드 기반
MagicPony: Learning Articulated 3D Animals in the Wild
We consider the problem of predicting the 3D shape, articulation, viewpoint, texture, and lighting of an articulated animal like a horse given a single test image as input. We present a new method, dubbed MagicPony, that…
Viewpoint EstimationShape Reconstruction from Thoracoscopic Images using Self-supervised Virtual Learning
Intraoperative shape reconstruction of organs from endoscopic camera images is a complex yet indispensable technique for image-guided surgery. To address the uncertainty in reconstructing entire shapes from single-viewpo…
Self-Supervised 3D Face Reconstruction via Conditional Estimation
We present a conditional estimation (CEST) framework to learn 3D facial parameters from 2D single-view images by self-supervised training from videos. CEST is based on the process of analysis by synthesis, where the 3D f…
3D Face ReconstructionDisentanglementFace ReconstructionPlanes vs. Chairs: Category-guided 3D shape learning without any 3D cues
We present a novel 3D shape reconstruction method which learns to predict an implicit 3D shape representation from a single RGB image. Our approach uses a set of single-view images of multiple object categories without v…
3D Shape Reconstruction3D Shape RepresentationMetric LearningSelf-Supervised 2D Image to 3D Shape Translation with Disentangled Representations
We present a framework to translate between 2D image views and 3D object shapes. Recent progress in deep learning enabled us to learn structure-aware representations from a scene. However, the existing literature assumes…
Image to 3DTranslation