Long-Term TalkingFace Generation via Motion-Prior Conditional Diffusion Model
Recent advances in conditional diffusion models have shown promise for generating realistic TalkingFace videos, yet challenges persist in achieving consistent head movement, synchronized facial expressions, and accurate lip synchronization over extended generations. To address these, we introduce the \textbf{M}otion-priors \textbf{C}onditional \textbf{D}iffusion \textbf{M}odel (\textbf{MCDM}), which utilizes both archived and current clip motion priors to enhance motion prediction and ensure temporal consistency. The model consists of three key elements: (1) an archived-clip motion-prior that incorporates historical frames and a reference frame to preserve identity and context; (2) a present-clip motion-prior diffusion model that captures multimodal causality for accurate predictions of head movements, lip sync, and expressions; and (3) a memory-efficient temporal attention mechanism that mitigates error accumulation by dynamically storing and updating motion features. We also release the \textbf{TalkingFace-Wild} dataset, a multilingual collection of over 200 hours of footage across 10 languages. Experimental results demonstrate the effectiveness of MCDM in maintaining identity and motion continuity for long-term TalkingFace generation. Code, models, and datasets will be publicly available.
Code (0)
등록된 구현이 없습니다.
Tasks
motion predictionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Talking-head Generation with Rhythmic Head Motion
When people deliver a speech, they naturally move heads, and this rhythmic head motion conveys prosodic information. However, generating a lip-synced video while moving head naturally is challenging. While remarkably suc…
Talking Head GenerationHybrid eye center localization using cascaded regression and hand-crafted model fitting
We propose a new cascaded regressor for eye center detection. Previous methods start from a face or an eye detector and use either advanced features or powerful regressors for eye center localization, but not both. Inste…
regressionLong-term Traffic Simulation with Interleaved Autoregressive Motion and Scenario Generation
An ideal traffic simulator replicates the realistic long-term point-to-point trip that a self-driving system experiences during deployment. Prior models and benchmarks focus on closed-loop motion simulation for initial a…
Scene GenerationCross-Conditioned Recurrent Networks for Long-Term Synthesis of Inter-Person Human Motion Interactions
Modeling dynamics of human motion is one of the most challenging sequence modeling problem, with diverse applications in animation industry, human-robot interaction, motion-based surveillance, etc. Available attempts to …
DecoderMotion Generationmotion predictionMotion Synthesis+1T2LM: Long-Term 3D Human Motion Generation from Multiple Sentences
In this paper, we address the challenging problem of long-term 3D human motion generation. Specifically, we aim to generate a long sequence of smoothly connected actions from a stream of multiple sentences (i.e., paragra…
Action GenerationDecoderMotion Generation