Learning 3D Object Shape and Layout without 3D Supervision
A 3D scene consists of a set of objects, each with a shape and a layout giving their position in space. Understanding 3D scenes from 2D images is an important goal, with applications in robotics and graphics. While there have been recent advances in predicting 3D shape and layout from a single image, most approaches rely on 3D ground truth for training which is expensive to collect at scale. We overcome these limitations and propose a method that learns to predict 3D shape and layout for objects without any ground truth shape or layout information: instead we rely on multi-view images with 2D supervision which can more easily be collected at scale. Through extensive experiments on 3D Warehouse, Hypersim, and ScanNet we demonstrate that our approach scales to large datasets of realistic images, and compares favorably to methods relying on 3D ground truth. On Hypersim and ScanNet where reliable 3D ground truth is not available, our approach outperforms supervised approaches trained on smaller and less diverse datasets.
Code (0)
등록된 구현이 없습니다.
Tasks
ObjectSimilar Papers 제목 키워드 기반
Learning 3D Scene Priors with 2D Supervision
Holistic 3D scene understanding entails estimation of both layout configuration and object geometry in a 3D environment. Recent works have shown advances in 3D scene estimation from various input modalities (e.g., images…
DecoderScene UnderstandingIterated learning for emergent systematicity in VQA
Although neural module networks have an architectural bias towards compositionality, they require gold standard layouts to generalize systematically in practice. When instead learning layouts and modules jointly, composi…
Question AnsweringSystematic GeneralizationVisual Question AnsweringVisual Question Answering (VQA)Recognizing Scenes from Novel Viewpoints
Humans can perceive scenes in 3D from a handful of 2D views. For AI agents, the ability to recognize a scene from any viewpoint given only a few images enables them to efficiently interact with the scene and its objects.…
Scene RecognitionHolistic 3D Scene Understanding from a Single Image with Implicit Representation
We present a new pipeline for holistic 3D scene understanding from a single image, which could predict object shapes, object poses, and scene layout. As it is a highly ill-posed problem, existing methods usually suffer f…
3D Object Detection3D Shape ReconstructionGraph Neural NetworkMonocular 3D Object Detection+5Weakly Supervised Learning of Multi-Object 3D Scene Decompositions Using Deep Shape Priors
Representing scenes at the granularity of objects is a prerequisite for scene understanding and decision making. We propose PriSMONet, a novel approach based on Prior Shape knowledge for learning Multi-Object 3D scene de…
Decision MakingScene UnderstandingWeakly-supervised Learning