Sync4D: Video Guided Controllable Dynamics for Physics-Based 4D Generation
In this work, we introduce a novel approach for creating controllable dynamics in 3D-generated Gaussians using casually captured reference videos. Our method transfers the motion of objects from reference videos to a variety of generated 3D Gaussians across different categories, ensuring precise and customizable motion transfer. We achieve this by employing blend skinning-based non-parametric shape reconstruction to extract the shape and motion of reference objects. This process involves segmenting the reference objects into motion-related parts based on skinning weights and establishing shape correspondences with generated target shapes. To address shape and temporal inconsistencies prevalent in existing methods, we integrate physical simulation, driving the target shapes with matched motion. This integration is optimized through a displacement loss to ensure reliable and genuine dynamics. Our approach supports diverse reference inputs, including humans, quadrupeds, and articulated objects, and can generate dynamics of arbitrary length, providing enhanced fidelity and applicability. Unlike methods heavily reliant on diffusion video generation models, our technique offers specific and high-quality motion transfer, maintaining both shape integrity and temporal consistency.
Code (0)
등록된 구현이 없습니다.
Tasks
Video GenerationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Audio-Synchronized Visual Animation
Current visual generation methods can produce high quality videos guided by texts. However, effectively controlling object dynamics remains a challenge. This work explores audio as a cue to generate temporally synchroniz…
PhysLayer: Language-Guided Layered Animation with Depth-Aware Physics
Existing image-to-video generation methods often produce physically implausible motions and lack precise control over object dynamics. While prior approaches have incorporated physics simulators, they remain confined to …
Computational EfficiencyScene Understanding3D ReconstructionVideo GenerationControlHair: Synergizing Physics Simulator and Video Diffusion for Controllable Dynamic Hair Rendering
Hair simulation and rendering are challenging due to complex strand dynamics, diverse material properties, and intricate light-hair interactions. Recent video diffusion models can generate high-quality videos, but they l…
Video GenerationPhysCtrl: Generative Physics for Controllable and Physics-Grounded Video Generation
Existing video generation models excel at producing photo-realistic videos from text or images, but often lack physical plausibility and 3D controllability. To overcome these limitations, we introduce PhysCtrl, a novel f…
Video GenerationSkyReels-Audio: Omni Audio-Conditioned Talking Portraits in Video Diffusion Transformers
The generation and editing of audio-conditioned talking portraits guided by multimodal inputs, including text, images, and videos, remains under explored. In this paper, we present SkyReels-Audio, a unified framework for…
Denoising