Disentangled 3D Scene Generation with Layout Learning

2024-02-26 · Dave Epstein, Ben Poole, Ben Mildenhall, Alexei A. Efros, Aleksander Holynski

We introduce a method to generate 3D scenes that are disentangled into their component objects. This disentanglement is unsupervised, relying only on the knowledge of a large pretrained text-to-image model. Our key insight is that objects can be discovered by finding parts of a 3D scene that, when rearranged spatially, still produce valid configurations of the same scene. Concretely, our method jointly optimizes multiple NeRFs from scratch - each representing its own object - along with a set of layouts that composite these objects into scenes. We then encourage these composited scenes to be in-distribution according to the image generator. We show that despite its simplicity, our approach successfully generates 3D scenes decomposed into individual objects, enabling new capabilities in text-to-3D content creation. For results and an interactive demo, see our project page at https://dave.ml/layoutlearning/

📄 PDF Abstract BibTeX arXiv:2402.16936

Code (0)

등록된 구현이 없습니다.

Tasks

Disentanglement Scene Generation Text to 3Dvalid

Methods 이 논문이 사용한 방법론

SET Dynamic Sparse Training method where weight mask is updated randomly periodically

Scene Graph Disentanglement and Composition for Generalizable Complex Image Generation

2024-10-01 · Yunnan Wang, Ziqiang Li, Zequn Zhang, Wenyao Zhang 외

There has been exciting progress in generating images from natural language or layout conditions. However, these methods struggle to faithfully reproduce complex scenes due to the insufficient modeling of multiple object…

Disentanglement Image Generation

LAW-Diffusion: Complex Scene Generation by Diffusion with Layouts

2023-08-13 · ICCV 2023 1 · BinBin Yang, Yi Luo, Ziliang Chen, Guangrun Wang 외

Thanks to the rapid development of diffusion models, unprecedented progress has been witnessed in image synthesis. Prior works mostly rely on pre-trained linguistic models, but a text is often too abstract to properly sp…

Image Generation Layout-to-Image Generation Object Scene Generation

SE-VGAE: Unsupervised Disentangled Representation Learning for Interpretable Architectural Layout Design Graph Generation

2024-06-25 · Jielin Chen, Rudi Stouffs

Despite the suitability of graphs for capturing the relational structures inherent in architectural layout designs, there is a notable dearth of research on interpreting architectural design space using graph-based repre…

Decoder Disentanglement Graph Generation Layout Design+1

Lighting-grounded Video Generation with Renderer-based Agent Reasoning

2026-04-09 · Ziqi Cai, Taoyu Yang, Zheng Chang, Si Li 외 arxiv

Diffusion models have achieved remarkable progress in video generation, but their controllability remains a major limitation. Key scene factors such as layout, lighting, and camera trajectory are often entangled or only …

Video-to-Video Synthesis Video Generation

Materialistic RIR: Material Conditioned Realistic RIR Generation

2026-04-22 · Mahnoor Fatima Saad, Sagnik Majumder, Kristen Grauman, Ziad Al-Halah arxiv

Rings like gold, thuds like wood! The sound we hear in a scene is shaped not only by the spatial layout of the environment but also by the materials of the objects and surfaces within it. For instance, a room with wooden…