EditYourself: Audio-Driven Generation and Manipulation of Talking Head Videos with Diffusion Transformers
Current generative video models excel at producing novel content from text and image prompts, but leave a critical gap in editing existing pre-recorded videos, where minor alterations to the spoken script require preserving motion, temporal coherence, speaker identity, and accurate lip synchronization. We introduce EditYourself, a DiT-based framework for audio-driven video-to-video (V2V) editing that enables transcript-based modification of talking head videos, including the seamless addition, removal, and retiming of visually spoken content. Building on a general-purpose video diffusion model, EditYourself augments its V2V capabilities with audio conditioning and region-aware, edit-focused training extensions. This enables precise lip synchronization and temporally coherent restructuring of existing performances via spatiotemporal inpainting, including the synthesis of realistic human motion in newly added segments, while maintaining visual fidelity and identity consistency over long durations. This work represents a foundational step toward generative video models as practical tools for professional video post-production.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Face2VoiceSync: Lightweight Face-Voice Consistency for Text-Driven Talking Face Generation
Recent studies in speech-driven talking face generation achieve promising results, but their reliance on fixed-driven speech limits further applications (e.g., face-voice mismatch). Thus, we extend the task to a more cha…
Talking Face GenerationFace AlignmentMEAD: A Large-scale Audio-visual Dataset for Emotional Talking-face Generation
The synthesis of natural emotional reactions is an essentialcriteria in vivid talking-face video generation. This criteria is nevertheless seldom taken into consideration in previous works due to the absence of a large-s…
Face GenerationTalking Face GenerationTalking Head GenerationVideo GenerationJoyGen: Audio-Driven 3D Depth-Aware Talking-Face Video Editing
Significant progress has been made in talking-face video generation research; however, precise lip-audio synchronization and high visual quality remain challenging in editing lip shapes based on input audio. This paper i…
3D ReconstructionFace GenerationMotion GenerationTalking Face Generation+2FTFDNet: Learning to Detect Talking Face Video Manipulation with Tri-Modality Interaction
DeepFake based digital facial forgery is threatening public media security, especially when lip manipulation has been used in talking face generation, and the difficulty of fake video detection is further improved. By on…
Face DetectionFace GenerationFace SwappingOptical Flow Estimation+1Exploring Talking Head Models With Adjacent Frame Prior for Speech-Preserving Facial Expression Manipulation
Speech-Preserving Facial Expression Manipulation (SPFEM) is an innovative technique aimed at altering facial expressions in images and videos while retaining the original mouth movements. Despite advancements, SPFEM stil…
Talking Head Generation