CARFF: Conditional Auto-encoded Radiance Field for 3D Scene Forecasting
We propose CARFF, a method for predicting future 3D scenes given past observations. Our method maps 2D ego-centric images to a distribution over plausible 3D latent scene configurations and predicts the evolution of hypothesized scenes through time. Our latents condition a global Neural Radiance Field (NeRF) to represent a 3D scene model, enabling explainable predictions and straightforward downstream planning. This approach models the world as a POMDP and considers complex scenarios of uncertainty in environmental states and dynamics. Specifically, we employ a two-stage training of Pose-Conditional-VAE and NeRF to learn 3D representations, and auto-regressively predict latent scene representations utilizing a mixture density network. We demonstrate the utility of our method in scenarios using the CARLA driving simulator, where CARFF enables efficient trajectory and contingency planning in complex multi-agent autonomous driving scenarios involving occlusions.
Code (0)
등록된 구현이 없습니다.
Tasks
Autonomous DrivingNeRFNeural RenderingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Audio-Driven Talking Face Generation with Blink Embedding and Hash Grid Landmarks Encoding
Dynamic Neural Radiance Fields (NeRF) have demonstrated considerable success in generating high-fidelity 3D models of talking portraits. Despite significant advancements in the rendering speed and generation quality, cha…
Talking Face GenerationPix2NeRF: Unsupervised Conditional p-GAN for Single Image to Neural Radiance Fields Translation
We propose a pipeline to generate Neural Radiance Fields (NeRF) of an object or a scene of a specific class, conditioned on a single input image. This is a challenging task, as training NeRF requires multiple views o…
3D-Aware Image SynthesisImage GenerationNeRFNovel View Synthesis+2Mesh2NeRF: Direct Mesh Supervision for Neural Radiance Field Representation and Generation
We present Mesh2NeRF, an approach to derive ground-truth radiance fields from textured meshes for 3D generation tasks. Many 3D generative approaches represent 3D scenes as radiance fields for training. Their ground-truth…
3D GenerationNeRFPix2NeRF: Unsupervised Conditional $π$-GAN for Single Image to Neural Radiance Fields Translation
We propose a pipeline to generate Neural Radiance Fields~(NeRF) of an object or a scene of a specific class, conditioned on a single input image. This is a challenging task, as training NeRF requires multiple views of th…
3D-Aware Image SynthesisImage GenerationNeRFNovel View Synthesis+2Conditional-Flow NeRF: Accurate 3D Modelling with Reliable Uncertainty Quantification
A critical limitation of current methods based on Neural Radiance Fields (NeRF) is that they are unable to quantify the uncertainty associated with the learned appearance and geometry of the scene. This information is pa…
Autonomous DrivingDecision MakingMedical DiagnosisNeRF+1