Reconstruct, Inpaint, Test-Time Finetune: Dynamic Novel-view Synthesis from Monocular Videos
We explore novel-view synthesis for dynamic scenes from monocular videos. Prior approaches rely on costly test-time optimization of 4D representations or do not preserve scene geometry when trained in a feed-forward manner. Our approach is based on three key insights: (1) covisible pixels (that are visible in both the input and target views) can be rendered by first reconstructing the dynamic 3D scene and rendering the reconstruction from the novel-views and (2) hidden pixels in novel views can be "inpainted" with feed-forward 2D video diffusion models. Notably, our video inpainting diffusion model (CogNVS) can be self-supervised from 2D videos, allowing us to train it on a large corpus of in-the-wild videos. This in turn allows for (3) CogNVS to be applied zero-shot to novel test videos via test-time finetuning. We empirically verify that CogNVS outperforms almost all prior art for novel-view synthesis of dynamic scenes from monocular videos.
Code (0)
등록된 구현이 없습니다.
Tasks
Video InpaintingSimilar Papers 제목 키워드 기반
Iterative Image Inpainting with Structural Similarity Mask for Anomaly Detection
Autoencoders have emerged as popular methods for unsupervised anomaly detection. Autoencoders trained on the normal data are expected to reconstruct only the normal features, allowing anomaly detection by thresholding re…
Anomaly DetectionImage InpaintingUnsupervised Anomaly DetectionEasyOmnimatte: Taming Pretrained Inpainting Diffusion Models for End-to-End Video Layered Decomposition
Existing video omnimatte methods typically rely on slow, multi-stage, or inference-time optimization pipelines that fail to fully exploit powerful generative priors, producing suboptimal decompositions. Our key insight i…
Video InpaintingDiffMVR: Diffusion-based Automated Multi-Guidance Video Restoration
In this work, we address a challenge in video inpainting: reconstructing occluded regions in dynamic, real-world scenarios. Motivated by the need for continuous human motion monitoring in healthcare settings, where facia…
Video InpaintingVideo RestorationOnline Adaptive Image Reconstruction (OnAIR) Using Dictionary Models
Sparsity and low-rank models have been popular for reconstructing images and videos from limited or corrupted measurements. Dictionary or transform learning methods are useful in applications such as denoising, inpaintin…
DenoisingImage ReconstructionVideo ReconstructionHierarchical Approach for Total Variation Digital Image Inpainting
The art of recovering an image from damage in an undetectable form is known as inpainting. The manual work of inpainting is most often a very time consuming process. Due to digitalization of this technique, it is automat…
Image Inpainting