HOSNeRF: Dynamic Human-Object-Scene Neural Radiance Fields from a Single Video
We introduce HOSNeRF, a novel 360{\deg} free-viewpoint rendering method that reconstructs neural radiance fields for dynamic human-object-scene from a single monocular in-the-wild video. Our method enables pausing the video at any frame and rendering all scene details (dynamic humans, objects, and backgrounds) from arbitrary viewpoints. The first challenge in this task is the complex object motions in human-object interactions, which we tackle by introducing the new object bones into the conventional human skeleton hierarchy to effectively estimate large object deformations in our dynamic human-object model. The second challenge is that humans interact with different objects at different times, for which we introduce two new learnable object state embeddings that can be used as conditions for learning our human-object representation and scene representation, respectively. Extensive experiments show that HOSNeRF significantly outperforms SOTA approaches on two challenging datasets by a large margin of 40% ~ 50% in terms of LPIPS. The code, data, and compelling examples of 360{\deg} free-viewpoint renderings from single videos will be released in https://showlab.github.io/HOSNeRF.
Code (0)
등록된 구현이 없습니다.
Tasks
Human-Object Interaction DetectionObjectSimilar Papers 제목 키워드 기반
CoGS: Compositional Dynamic Human-Object Scenes Gaussian Splatting from Monocular Video
Reconstructing dynamic human--object interaction scenes from monocular video is difficult because the human, manipulated object, and background obey different motion models while sharing the same pixels. Existing dynamic…
STaR: Self-supervised Tracking and Reconstruction of Rigid Objects in Motion with Neural Rendering
We present STaR, a novel method that performs Self-supervised Tracking and Reconstruction of dynamic scenes with rigid motion from multi-view RGB videos without any manual annotation. Recent work has shown that neural ne…
Neural RenderingObjectNeRF2Real: Sim2real Transfer of Vision-guided Bipedal Motion Skills using Neural Radiance Fields
We present a system for applying sim2real approaches to "in the wild" scenes with realistic visuals, and to policies which rely on active perception using RGB cameras. Given a short video of a static scene collected usin…
NeRFNovel View SynthesisFLAME-in-NeRF: Neural control of Radiance Fields for Free View Face Animation
This paper presents a neural rendering method for controllable portrait video synthesis.Recent advances in volumetric neural rendering, such as neural radiance fields (NeRF), have enabled the photorealistic novel view sy…
NeRFNeural RenderingNovel View SynthesisNeRSemble: Multi-view Radiance Field Reconstruction of Human Heads
We focus on reconstructing high-fidelity radiance fields of human heads, capturing their animations over time, and synthesizing re-renderings from novel viewpoints at arbitrary time steps. To this end, we propose a new m…