Realistic Speech-Driven Facial Animation with GANs

2019-06-14 · Konstantinos Vougioukas, Stavros Petridis, Maja Pantic

Speech-driven facial animation is the process that automatically synthesizes talking characters based on speech signals. The majority of work in this domain creates a mapping from audio features to visual features. This approach often requires post-processing using computer graphics techniques to produce realistic albeit subject dependent results. We present an end-to-end system that generates videos of a talking head, using only a still image of a person and an audio clip containing speech, without relying on handcrafted intermediate features. Our method generates videos which have (a) lip movements that are in sync with the audio and (b) natural facial expressions such as blinks and eyebrow movements. Our temporal GAN uses 3 discriminators focused on achieving detailed frames, audio-visual synchronization, and realistic expressions. We quantify the contribution of each component in our model using an ablation study and we provide insights into the latent representation of the model. The generated videos are evaluated based on sharpness, reconstruction quality, lip-reading accuracy, synchronization as well as their ability to generate natural blinks.

📄 PDF Abstract BibTeX arXiv:1906.06337

Code (0)

등록된 구현이 없습니다.

Tasks

Audio-Visual SynchronizationLip Reading

Methods 이 논문이 사용한 방법론

Convolution A convolution is a type of matrix operation, consisting of a kernel, a small matrix of weights, that slides over input data performing element-wise multiplication with the…

Dogecoin Customer Service Number +1-833-534-1729 설명 없음

Speech-driven Facial Animation using Cascaded GANs for Learning of Motion and Texture

2020-08-01 · ECCV 2020 8 · Dipanjan Das, Sandika Biswas, Sanjana Sinha, Brojeshwar Bhowmick

Speech-driven facial animation methods should produce accurate and realistic lip motions with natural expressions and realistic texture portraying target-specific facial characteristics. Moreover, the methods should also…

Meta-Learning

End-to-End Speech-Driven Facial Animation with Temporal GANs

2018-05-23 · Konstantinos Vougioukas, Stavros Petridis, Maja Pantic

Speech-driven facial animation is the process which uses speech signals to automatically synthesize a talking character. The majority of work in this domain creates a mapping from audio features to visual features. This …

Lip Reading

Identity-Preserving Realistic Talking Face Generation

2020-05-25 · Sanjana Sinha, Sandika Biswas, Brojeshwar Bhowmick

Speech-driven facial animation is useful for a variety of applications such as telepresence, chatbots, etc. The necessary attributes of having a realistic face animation are 1) audio-visual synchronization (2) identity p…

Audio-Visual Synchronization Face Generation Image Reconstruction Talking Face Generation

DF-3DFace: One-to-Many Speech Synchronized 3D Face Animation with Diffusion

2023-08-23 · Se Jin Park, Joanna Hong, Minsu Kim, Yong Man Ro

Speech-driven 3D facial animation has gained significant attention for its ability to create realistic and expressive facial animations in 3D space based on speech. Learning-based methods have shown promising progress in…

3D Face Animation

Breathing Life into Faces: Speech-driven 3D Facial Animation with Natural Head Pose and Detailed Shape

2023-10-31 · Wei Zhao, Yijun Wang, Tianyu He, Lianying Yin 외

The creation of lifelike speech-driven 3D facial animation requires a natural and precise synchronization between audio input and facial expressions. However, existing works still fail to render shapes with flexible head…