MaskRenderer: 3D-Infused Multi-Mask Realistic Face Reenactment

2023-09-10 · Tina Behrouzi, Atefeh Shahroudnejad, Payam Mousavi

We present a novel end-to-end identity-agnostic face reenactment system, MaskRenderer, that can generate realistic, high fidelity frames in real-time. Although recent face reenactment works have shown promising results, there are still significant challenges such as identity leakage and imitating mouth movements, especially for large pose changes and occluded faces. MaskRenderer tackles these problems by using (i) a 3DMM to model 3D face structure to better handle pose changes, occlusion, and mouth movements compared to 2D representations; (ii) a triplet loss function to embed the cross-reenactment during training for better identity preservation; and (iii) multi-scale occlusion, improving inpainting and restoring missing areas. Comprehensive quantitative and qualitative experiments conducted on the VoxCeleb1 test set, demonstrate that MaskRenderer outperforms state-of-the-art models on unseen faces, especially when the Source and Driving identities are very different.

📄 PDF Abstract BibTeX arXiv:2309.05095

Code (0)

등록된 구현이 없습니다.

Tasks

Face ReenactmentTriplet

Methods 이 논문이 사용한 방법론

Triplet Loss The goal of Triplet loss, in the context of Siamese Networks, is to maximize the joint probability among all score-pairs i.e. the product of all probabilities. By using its…

Inpainting Train a convolutional neural network to generate the contents of an arbitrary image region conditioned on its surroundings.

On Hallucinating Context and Background Pixels from a Face Mask using Multi-scale GANs

2018-11-17 · Sandipan Banerjee, Walter J. Scheirer, Kevin W. Bowyer, Patrick J. Flynn

We propose a multi-scale GAN model to hallucinate realistic context (forehead, hair, neck, clothes) and background pixels automatically from a single input face mask. Instead of swapping a face on to an existing picture,…

Face Swapping Facial Inpainting Hallucination

Vision-Infused Deep Audio Inpainting

2019-10-24 · ICCV 2019 10 · Hang Zhou, Ziwei Liu, Xudong Xu, Ping Luo 외

Multi-modality perception is essential to develop interactive intelligence. In this work, we consider a new task of visual information-infused audio inpainting, \ie synthesizing missing audio segments that correspond to …

Audio inpainting Image Inpainting

A realistic approach to generate masked faces applied on two novel masked face recognition data sets

2021-09-03 · Tudor Mare, Georgian Duta, Mariana-Iuliana Georgescu, Adrian Sandru 외

The COVID-19 pandemic raises the problem of adapting face recognition systems to the new reality, where people may wear surgical masks to cover their noses and mouths. Traditional data sets (e.g., CelebA, CASIA-WebFace) …

Face Recognition

KiNETGAN: Enabling Distributed Network Intrusion Detection through Knowledge-Infused Synthetic Data Generation

2024-05-26 · Anantaa Kotal, Brandon Luton, Anupam Joshi

In the realm of IoT/CPS systems connected over mobile networks, traditional intrusion detection methods analyze network traffic across multiple devices using anomaly detection techniques to flag potential security threat…

Anomaly Detection Generative Adversarial Network Intrusion Detection Network Intrusion Detection+1

MLFW: A Database for Face Recognition on Masked Faces

2021-09-13 · Chengrui Wang, Han Fang, Yaoyao Zhong, Weihong Deng

As more and more people begin to wear masks due to current COVID-19 pandemic, existing face recognition systems may encounter severe performance degradation when recognizing masked faces. To figure out the impact of mask…