FLAME-in-NeRF : Neural control of Radiance Fields for Free View Face Animation
This paper presents a neural rendering method for controllable portrait video synthesis. Recent advances in volumetric neural rendering, such as neural radiance fields (NeRF), has enabled the photorealistic novel view synthesis of static scenes with impressive results. However, modeling dynamic and controllable objects as part of a scene with such scene representations is still challenging. In this work, we design a system that enables both novel view synthesis for portrait video, including the human subject and the scene background, and explicit control of the facial expressions through a low-dimensional expression representation. We leverage the expression space of a 3D morphable face model (3DMM) to represent the distribution of human facial expressions, and use it to condition the NeRF volumetric function. Furthermore, we impose a spatial prior brought by 3DMM fitting to guide the network to learn disentangled control for scene appearance and facial actions. We demonstrate the effectiveness of our method on free view synthesis of portrait videos with expression controls. To train a scene, our method only requires a short video of a subject captured by a mobile device.
Code (0)
등록된 구현이 없습니다.
Tasks
Face ModelNeRFNeural RenderingNovel View SynthesisSimilar Papers 제목 키워드 기반
FLAME-in-NeRF: Neural control of Radiance Fields for Free View Face Animation
This paper presents a neural rendering method for controllable portrait video synthesis.Recent advances in volumetric neural rendering, such as neural radiance fields (NeRF), have enabled the photorealistic novel view sy…
NeRFNeural RenderingNovel View SynthesisNeRFlame: FLAME-based conditioning of NeRF for 3D face rendering
Traditional 3D face models are based on mesh representations with texture. One of the most important models is FLAME (Faces Learned with an Articulated Model and Expressions), which produces meshes of human faces that ar…
NeRFHigh-Fidelity Eye Animatable Neural Radiance Fields for Human Face
Face rendering using neural radiance fields (NeRF) is a rapidly developing research area in computer vision. While recent methods primarily focus on controlling facial attributes such as identity and expression, they oft…
Face ModelGaze EstimationNeRFEmbedded Representation Learning Network for Animating Styled Video Portrait
The talking head generation recently attracted considerable attention due to its widespread application prospects, especially for digital avatars and 3D animation design. Inspired by this practical demand, several works …
NeRFRepresentation LearningTalking Head GenerationHallucinated Neural Radiance Fields in the Wild
Neural Radiance Fields (NeRF) has recently gained popularity for its impressive novel view synthesis ability. This paper studies the problem of hallucinated NeRF: i.e., recovering a realistic NeRF at a different time of …
HallucinationNeRFNovel View Synthesis