Controllable Talking Face Generation by Implicit Facial Keypoints Editing
Audio-driven talking face generation has garnered significant interest within the domain of digital human research. Existing methods are encumbered by intricate model architectures that are intricately dependent on each other, complicating the process of re-editing image or video inputs. In this work, we present ControlTalk, a talking face generation method to control face expression deformation based on driven audio, which can construct the head pose and facial expression including lip motion for both single image or sequential video inputs in a unified manner. By utilizing a pre-trained video synthesis renderer and proposing the lightweight adaptation, ControlTalk achieves precise and naturalistic lip synchronization while enabling quantitative control over mouth opening shape. Our experiments show that our method is superior to state-of-the-art performance on widely used benchmarks, including HDTF and MEAD. The parameterized adaptation demonstrates remarkable generalization capabilities, effectively handling expression deformation across same-ID and cross-ID scenarios, and extending its utility to out-of-domain portraits, regardless of languages. Code is available at https://github.com/NetEase-Media/ControlTalk.
Code (1)
Tasks
Face GenerationTalking Face GenerationSimilar Papers 제목 키워드 기반
That's What I Said: Fully-Controllable Talking Face Generation
The goal of this paper is to synthesise talking faces with controllable facial motions. To achieve this goal, we propose two key ideas. The first is to establish a canonical space where every face has the same motion pat…
Face GenerationNavigateTalking Face GenerationDynamic Neural Textures: Generating Talking-Face Videos with Continuously Controllable Expressions
Recently, talking-face video generation has received considerable attention. So far most methods generate results with neutral expressions or expressions that are implicitly determined by neural networks in an uncontroll…
Video GenerationGeometry-guided Emotion Modulation for Controllable and Photorealistic Emotional Talking Face Generation
Audio-driven emotional talking face generation aims to synthesize realistic videos with expressive facial dynamics. However, existing methods struggle to balance controllability and visual fidelity. Although implicit rep…
Talking Face GenerationContinuous ControlOPT: One-shot Pose-Controllable Talking Head Generation
One-shot talking head generation produces lip-sync talking heads based on arbitrary audio and one source face. To guarantee the naturalness and realness, recent methods propose to achieve free pose control instead of sim…
DisentanglementTalking Head GenerationPlaymate: Flexible Control of Portrait Animation via 3D-Implicit Space Guided Diffusion
Recent diffusion-based talking face generation models have demonstrated impressive potential in synthesizing videos that accurately match a speech audio clip with a given reference identity. However, existing approaches …
AttributeDisentanglementFace GenerationPortrait Animation+1