LatentFusion: End-to-End Differentiable Reconstruction and Rendering for Unseen Object Pose Estimation
Current 6D object pose estimation methods usually require a 3D model for each object. These methods also require additional training in order to incorporate new objects. As a result, they are difficult to scale to a large number of objects and cannot be directly applied to unseen objects. We propose a novel framework for 6D pose estimation of unseen objects. We present a network that reconstructs a latent 3D representation of an object using a small number of reference views at inference time. Our network is able to render the latent 3D representation from arbitrary views. Using this neural renderer, we directly optimize for pose given an input image. By training our network with a large number of 3D shapes for reconstruction and rendering, our network generalizes well to unseen objects. We present a new dataset for unseen object pose estimation--MOPED. We evaluate the performance of our method for unseen object pose estimation on MOPED as well as the ModelNet and LINEMOD datasets. Our method performs competitively to supervised methods that are trained on those objects. Code and data is available at https://keunhong.com/publications/latentfusion/.
Code (1)
Tasks
6D Pose Estimation6D Pose Estimation using RGBObjectPose EstimationSimilar Papers 제목 키워드 기반
Differentiable Inverse Graphics for Zero-shot Scene Reconstruction and Robot Grasping
Operating effectively in novel real-world environments requires robotic systems to estimate and interact with previously unseen objects. Current state-of-the-art models address this challenge by using large amounts of tr…
Pose EstimationMonocular Differentiable Rendering for Self-Supervised 3D Object Detection
3D object detection from monocular images is an ill-posed problem due to the projective entanglement of depth and scale. To overcome this ambiguity, we present a novel self-supervised method for textured 3D shape reconst…
3D Object Detection3D Object Detection From Monocular Images3D Shape ReconstructionDepth Estimation+5SDFDiff: Differentiable Rendering of Signed Distance Fields for 3D Shape Optimization
We propose SDFDiff, a novel approach for image-based shape optimization using differentiable rendering of 3D shapes represented by signed distance functions (SDFs). Compared to other representations, SDFs have the advant…
3D ReconstructionMulti-View 3D ReconstructionSingle-View 3D ReconstructionStyle Agnostic 3D Reconstruction via Adversarial Style Transfer
Reconstructing the 3D geometry of an object from an image is a major challenge in computer vision. Recently introduced differentiable renderers can be leveraged to learn the 3D geometry of objects from 2D images, but tho…
3D geometry3D Object Reconstruction3D ReconstructionDomain Adaptation+3Differentiable Rendering for Synthetic Aperture Radar Imagery
There is rising interest in differentiable rendering, which allows explicitly modeling geometric priors and constraints in optimization pipelines using first-order methods such as backpropagation. Incorporating such doma…
3D Object ReconstructionNeural RenderingObject Reconstruction