Self-Supervised Monocular 3D Face Reconstruction by Occlusion-Aware Multi-view Geometry Consistency
Recent learning-based approaches, in which models are trained by single-view images have shown promising results for monocular 3D face reconstruction, but they suffer from the ill-posed face pose and depth ambiguity issue. In contrast to previous works that only enforce 2D feature constraints, we propose a self-supervised training architecture by leveraging the multi-view geometry consistency, which provides reliable constraints on face pose and depth estimation. We first propose an occlusion-aware view synthesis method to apply multi-view geometry consistency to self-supervised learning. Then we design three novel loss functions for multi-view consistency, including the pixel consistency loss, the depth consistency loss, and the facial landmark-based epipolar loss. Our method is accurate and robust, especially under large variations of expressions, poses, and illumination conditions. Comprehensive experiments on the face alignment and 3D face reconstruction benchmarks have demonstrated superiority over state-of-the-art methods. Our code and model are released in https://github.com/jiaxiangshang/MGCNet.
Code (1)
Tasks
3D Face ReconstructionDepth EstimationFace AlignmentFace ReconstructionSelf-Supervised LearningSimilar Papers 제목 키워드 기반
Self-Supervised Robustifying Guidance for Monocular 3D Face Reconstruction
Despite the recent developments in 3D Face Reconstruction from occluded and noisy face images, the performance is still unsatisfactory. Moreover, most existing methods rely on additional dependencies, posing numerous con…
3D Face ReconstructionFace ReconstructionOcclusion-Aware Self-Supervised Monocular Depth Estimation for Weak-Texture Endoscopic Images
We propose a self-supervised monocular depth estimation network tailored for endoscopic scenes, aiming to infer depth within the gastrointestinal tract from monocular images. Existing methods, though accurate, typically …
Data AugmentationDepth EstimationMonocular Depth EstimationSemantic SegmentationS2F2: Self-Supervised High Fidelity Face Reconstruction from Monocular Image
We present a novel face reconstruction method capable of reconstructing detailed face geometry, spatially varying face reflectance from a single monocular image. We build our work upon the recent advances of DNN-based au…
3D Face ReconstructionFace ReconstructionSelf-Supervised LearningVocal Bursts Intensity PredictionSynthesizing Light Field Video from Monocular Video
The hardware challenges associated with light-field(LF) imaging has made it difficult for consumers to access its benefits like applications in post-capture focus and aperture control. Learning-based techniques which sol…
Self-Supervised LearningVideo ReconstructionTowards High Fidelity Monocular Face Reconstruction with Rich Reflectance using Self-supervised Learning and Ray Tracing
Robust face reconstruction from monocular image in general lighting conditions is challenging. Methods combining deep neural network encoders with differentiable rendering have opened up the path for very fast monocular …
3D Face ReconstructionFace ReconstructionMonocular ReconstructionSelf-Supervised Learning