RenderDiffusion: Image Diffusion for 3D Reconstruction, Inpainting and Generation
Diffusion models currently achieve state-of-the-art performance for both conditional and unconditional image generation. However, so far, image diffusion models do not support tasks required for 3D understanding, such as view-consistent 3D generation or single-view object reconstruction. In this paper, we present RenderDiffusion, the first diffusion model for 3D generation and inference, trained using only monocular 2D supervision. Central to our method is a novel image denoising architecture that generates and renders an intermediate three-dimensional representation of a scene in each denoising step. This enforces a strong inductive structure within the diffusion process, providing a 3D consistent representation while only requiring 2D supervision. The resulting 3D representation can be rendered from any view. We evaluate RenderDiffusion on FFHQ, AFHQ, ShapeNet and CLEVR datasets, showing competitive performance for generation of 3D scenes and inference of 3D scenes from 2D images. Additionally, our diffusion-based approach allows us to use 2D inpainting to edit 3D scenes.
Code (1)
Tasks
3D Generation3D ReconstructionDenoisingImage DenoisingImage GenerationObject ReconstructionUnconditional Image GenerationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Learning Sparse Masks for Diffusion-based Image Inpainting
Diffusion-based inpainting is a powerful tool for the reconstruction of images from sparse data. Its quality strongly depends on the choice of known data. Optimising their spatial location -- the inpainting mask -- is ch…
Image CompressionImage InpaintingInpaint3D: 3D Scene Content Generation using 2D Inpainting Diffusion
This paper presents a novel approach to inpainting 3D regions of a scene, given masked multi-view images, by distilling a 2D diffusion model into a learned 3D scene representation (e.g. a NeRF). Unlike 3D generative meth…
3D InpaintingNeRFObjectEnd4: End-to-end Denoising Diffusion for Diffusion-Based Inpainting Detection
The powerful generative capabilities of diffusion models have significantly advanced the field of image synthesis, enhancing both full image generation and inpainting-based image editing. Despite their remarkable advance…
Image GenerationImage EditingHow Do Inpainting Artifacts Propagate to Language?
We study how visual artifacts introduced by diffusion-based inpainting affect language generation in vision-language models. We use a two-stage diagnostic setup in which masked image regions are reconstructed and then pr…
Reconstruct or Generate: Exploring the Spectrum of Generative Modeling for Cardiac MRI
In medical imaging, generative models are increasingly relied upon for two distinct but equally critical tasks: reconstruction, where the goal is to restore medical imaging (usually inverse problems like inpainting or su…
Image InpaintingImage Generation