Arbitrary Talking Face Generation via Attentional Audio-Visual Coherence Learning
Talking face generation aims to synthesize a face video with precise lip synchronization as well as a smooth transition of facial motion over the entire video via the given speech clip and facial image. Most existing methods mainly focus on either disentangling the information in a single image or learning temporal information between frames. However, cross-modality coherence between audio and video information has not been well addressed during synthesis. In this paper, we propose a novel arbitrary talking face generation framework by discovering the audio-visual coherence via the proposed Asymmetric Mutual Information Estimator (AMIE). In addition, we propose a Dynamic Attention (DA) block by selectively focusing the lip area of the input image during the training stage, to further enhance lip synchronization. Experimental results on benchmark LRW dataset and GRID dataset transcend the state-of-the-art methods on prevalent metrics with robust high-resolution synthesizing on gender and pose variations.
Code (0)
등록된 구현이 없습니다.
Tasks
Face GenerationTalking Face GenerationSimilar Papers 제목 키워드 기반
Talking Face Generation by Adversarially Disentangled Audio-Visual Representation
Talking face generation aims to synthesize a sequence of face images that correspond to a clip of speech. This is a challenging task because face appearance variation and semantics of speech are coupled together in the s…
Face GenerationLip ReadingRetrievalTalking Face Generation+1EAMM: One-Shot Emotional Talking Face via Audio-Based Emotion-Aware Motion Model
Although significant progress has been made to audio-driven talking face generation, existing methods either neglect facial emotion or cannot be applied to arbitrary subjects. In this paper, we propose the Emotion-Aware …
Face GenerationTalking Face GenerationOPT: One-shot Pose-Controllable Talking Head Generation
One-shot talking head generation produces lip-sync talking heads based on arbitrary audio and one source face. To guarantee the naturalness and realness, recent methods propose to achieve free pose control instead of sim…
DisentanglementTalking Head GenerationFlow-Guided One-Shot Talking Face Generation With a High-Resolution Audio-Visual Dataset
One-shot talking face generation should synthesize high visual quality facial videos with reasonable animations of expression and head pose, and just utilize arbitrary driving audio and arbitrary single face image as…
Face GenerationTalking Face GenerationGeneFace: Generalized and High-Fidelity Audio-Driven 3D Talking Face Synthesis
Generating photo-realistic video portrait with arbitrary speech audio is a crucial problem in film-making and virtual reality. Recently, several works explore the usage of neural radiance field in this task to improve 3D…
Face GenerationLip ReadingNeRFTalking Face Generation+1