BulletGen: Improving 4D Reconstruction with Bullet-Time Generation
Transforming casually captured, monocular videos into fully immersive dynamic experiences is a highly ill-posed task, and comes with significant challenges, e.g., reconstructing unseen regions, and dealing with the ambiguity in monocular depth estimation. In this work we introduce BulletGen, an approach that takes advantage of generative models to correct errors and complete missing information in a Gaussian-based dynamic scene representation. This is done by aligning the output of a diffusion-based video generation model with the 4D reconstruction at a single frozen "bullet-time" step. The generated frames are then used to supervise the optimization of the 4D Gaussian model. Our method seamlessly blends generative content with both static and dynamic scene components, achieving state-of-the-art results on both novel-view synthesis, and 2D/3D tracking tasks.
Code (0)
등록된 구현이 없습니다.
Tasks
4D reconstructionDepth EstimationMonocular Depth EstimationNovel View SynthesisVideo GenerationSimilar Papers 제목 키워드 기반
Feed-Forward Bullet-Time Reconstruction of Dynamic Scenes from Monocular Videos
Recent advancements in static feed-forward scene reconstruction have demonstrated significant progress in high-quality novel view synthesis. However, these models often struggle with generalizability across diverse envir…
Novel View SynthesisiButter: Neural Interactive Bullet Time Generator for Human Free-viewpoint Rendering
Generating ``bullet-time'' effects of human free-viewpoint videos is critical for immersive visual effects and VR/AR experience. Recent neural advances still lack the controllable and interactive bullet-time design abili…
NeRFVideo GenerationSummarize, Outline, and Elaborate: Long-Text Generation via Hierarchical Supervision from Extractive Summaries
The difficulty of generating coherent long texts lies in the fact that existing models overwhelmingly focus on predicting local words, and cannot make high level plans on what to generate or capture the high-level discou…
Text GenerationKeiki: Towards Realistic Danmaku Generation via Sequential GANs
Search-based procedural content generation methods have recently been introduced for the autonomous creation of bullet hell games. Search-based methods, however, can hardly model patterns of danmakus -- the bullet hell s…
DiversityTime SeriesTime Series AnalysisMulti-view reconstruction of bullet time effect based on improved NSFF model
Bullet time is a type of visual effect commonly used in film, television and games that makes time seem to slow down or stop while still preserving dynamic details in the scene. It usually requires multiple sets of camer…
Neural RenderingOptical Flow Estimation