Navigating Large-Pose Challenge for High-Fidelity Face Reenactment with Video Diffusion Model
Face reenactment aims to generate realistic talking head videos by transferring motion from a driving video to a static source image while preserving the source identity. Although existing methods based on either implicit or explicit keypoints have shown promise, they struggle with large pose variations due to warping artifacts or the limitations of coarse facial landmarks. In this paper, we present the Face Reenactment Video Diffusion model (FRVD), a novel framework for high-fidelity face reenactment under large pose changes. Our method first employs a motion extractor to extract implicit facial keypoints from the source and driving images to represent fine-grained motion and to perform motion alignment through a warping module. To address the degradation introduced by warping, we introduce a Warping Feature Mapper (WFM) that maps the warped source image into the motion-aware latent space of a pretrained image-to-video (I2V) model. This latent space encodes rich priors of facial dynamics learned from large-scale video data, enabling effective warping correction and enhancing temporal coherence. Extensive experiments show that FRVD achieves superior performance over existing methods in terms of pose accuracy, identity preservation, and visual quality, especially in challenging scenarios with extreme pose variations.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Frozen Layers: Memory-efficient Many-fidelity Hyperparameter Optimization
As model sizes grow, finding efficient and cost-effective hyperparameter optimization (HPO) methods becomes increasingly crucial for deep learning pipelines. While multi-fidelity HPO (MF-HPO) trades off computational res…
GPUHyperparameter OptimizationStructSynth: Leveraging LLMs for Structure-Aware Tabular Data Synthesis in Low-Data Regimes
The application of machine learning on tabular data in specialized domains is severely limited by data scarcity. While generative models offer a solution, traditional methods falter in low-data regimes, and recent Large …
Anchor-Controlled Generative Adversarial Network for High-Fidelity Electromagnetic and Structurally Diverse Metasurface Design
Metasurfaces, capable of manipulating light at subwavelength scales, hold great potential for advancing optoelectronic applications. Generative models, particularly Generative Adversarial Networks (GANs), offer a promisi…
DiversityGenerative Adversarial NetworkNaP-Control: Navigating Diffusion Prior for Versatile and Fast Character Control
Achieving precise, versatile whole-body character control in physics-based animation remains challenging. Recent diffusion-based policies generate rich and expressive motions but typically rely on gradient-based test-tim…
Reinforcement LearningMITRA: An AI Assistant for Knowledge Retrieval in Physics Collaborations
Large-scale scientific collaborations, such as the Compact Muon Solenoid (CMS) at CERN, produce a vast and ever-growing corpus of internal documentation. Navigating this complex information landscape presents a significa…