Temporal-MPI: Enabling Multi-Plane Images for Dynamic Scene Modelling via Temporal Basis Learning
Novel view synthesis of static scenes has achieved remarkable advancements in producing photo-realistic results. However, key challenges remain for immersive rendering of dynamic scenes. One of the seminal image-based rendering method, the multi-plane image (MPI), produces high novel-view synthesis quality for static scenes. But modelling dynamic contents by MPI is not studied. In this paper, we propose a novel Temporal-MPI representation which is able to encode the rich 3D and dynamic variation information throughout the entire video as compact temporal basis and coefficients jointly learned. Time-instance MPI for rendering can be generated efficiently using mini-seconds by linear combinations of temporal basis and coefficients from Temporal-MPI. Thus novel-views at arbitrary time-instance will be able to be rendered via Temporal-MPI in real-time with high visual quality. Our method is trained and evaluated on Nvidia Dynamic Scene Dataset. We show that our proposed Temporal- MPI is much faster and more compact compared with other state-of-the-art dynamic scene modelling methods.
Code (0)
등록된 구현이 없습니다.
Tasks
Novel View SynthesisSimilar Papers 제목 키워드 기반
An Aligned Multi-Temporal Multi-Resolution Satellite Image Dataset for Change Detection Research
This paper presents an aligned multi-temporal and multi-resolution satellite image dataset for research in change detection. We expect our dataset to be useful to researchers who want to fuse information from multiple sa…
Change DetectionSHaDe: Compact and Consistent Dynamic 3D Reconstruction via Tri-Plane Deformation and Latent Diffusion
We present a novel framework for dynamic 3D scene reconstruction that integrates three key components: an explicit tri-plane deformation field, a view-conditioned canonical radiance field with spherical harmonics (SH) at…
3D Reconstruction3D Scene ReconstructionDenoisingImage ReconstructionSurMo: Surface-based 4D Motion Modeling for Dynamic Human Rendering
Dynamic human rendering from video sequences has achieved remarkable progress by formulating the rendering as a mapping from static poses to human images. However, existing methods focus on the human appearance reconstru…
Generalizable Novel View SynthesisNovel View SynthesisFast View Synthesis of Casual Videos with Soup-of-Planes
Novel view synthesis from an in-the-wild video is difficult due to challenges like scene dynamics and lack of parallax. While existing methods have shown promising results with implicit neural radiance fields, they are s…
Novel View SynthesisReal-time prediction of workplane illuminance distribution for daylight-linked controls using non-intrusive multimodal deep learning
Daylight-linked controls (DLCs) have significant potential for energy savings in buildings, especially when abundant daylight is available and indoor workplane illuminance can be accurately predicted in real time. Most e…
Multimodal Deep Learning