Bringing Objects to Life: 4D generation from 3D objects
Recent advancements in generative modeling now enable the creation of 4D content (moving 3D objects) controlled with text prompts. 4D generation has large potential in applications like virtual worlds, media, and gaming, but existing methods provide limited control over the appearance and geometry of generated content. In this work, we introduce a method for animating user-provided 3D objects by conditioning on textual prompts to guide 4D generation, enabling custom animations while maintaining the identity of the original object. We first convert a 3D mesh into a ``static" 4D Neural Radiance Field (NeRF) that preserves the visual attributes of the input object. Then, we animate the object using an Image-to-Video diffusion model driven by text. To improve motion realism, we introduce an incremental viewpoint selection protocol for sampling perspectives to promote lifelike movement and a masked Score Distillation Sampling (SDS) loss, which leverages attention maps to focus optimization on relevant regions. We evaluate our model in terms of temporal coherence, prompt adherence, and visual fidelity and find that our method outperforms baselines that are based on other approaches, achieving up to threefold improvements in identity preservation measured using LPIPS scores, and effectively balancing visual quality with dynamic content.
Code (0)
등록된 구현이 없습니다.
Tasks
NeRFMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
A Novel Design of a Parallel Machine Learnt Generational Garbage Collector
The Generational Garbage collection involves organizing the heap into different divisions of memory space in-order to filter long-lived objects from short-lived objects through moving the surviving object of each generat…
ObjectREArtGS: Reconstructing and Generating Articulated Objects via 3D Gaussian Splatting with Geometric and Motion Constraints
Articulated objects, as prevalent entities in human life, their 3D representations play crucial roles across various applications. However, achieving both high-fidelity textured surface reconstruction and dynamic generat…
Surface ReconstructionOpen microscopy in the life sciences: Quo Vadis?
Light microscopy allows observing cellular features and objects with sub-micrometer resolution. As such, light microscopy has been playing a fundamental role in the life sciences for more than a hundred years. Fueled by …
Automatic dataset generation for specific object detection
In the past decade, object detection tasks are defined mostly by large public datasets. However, building object detection datasets is not scalable due to inefficient image collecting and labeling. Furthermore, most labe…
Dataset GenerationObjectobject-detectionObject DetectionYou Only Erase Once: Erasing Anything without Bringing Unexpected Content
We present YOEO, an approach for object erasure. Unlike recent diffusion-based methods which struggle to erase target objects without generating unexpected content within the masked regions due to lack of sufficient pair…