Omnimatte3D: Associating Objects and Their Effects in Unconstrained Monocular Video
We propose a method to decompose a video into a background and a set of foreground layers, where the background captures stationary elements while the foreground layers capture moving objects along with their associated effects (e.g. shadows and reflections). Our approach is designed for unconstrained monocular videos, with arbitrary camera and object motion. Prior work that tackles this problem assumes that the video can be mapped onto a fixed 2D canvas, severely limiting the possible space of camera motion. Instead, our method applies recent progress in monocular camera pose and depth estimation to create a full, RGBD video layer for the background, along with a video layer for each foreground object. To solve the underconstrained decomposition problem, we propose a new loss formulation based on multi-view consistency. We test our method on challenging videos with complex camera motion and show significant qualitative improvement over current approaches.
Code (0)
등록된 구현이 없습니다.
Tasks
Depth EstimationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Omnimatte: Associating Objects and Their Effects in Video
Computer vision is increasingly effective at segmenting objects in images and videos; however, scene effects related to the objects -- shadows, reflections, generated smoke, etc -- are typically overlooked. Identifying s…
OmnimatteZero: Training-free Real-time Omnimatte with Pre-trained Video Diffusion Models
Omnimatte aims to decompose a given video into semantically meaningful layers, including the background and individual objects along with their associated effects, such as shadows and reflections. Existing methods often …
Image InpaintingObjectOmnimatteRF: Robust Omnimatte with 3D Background Modeling
Video matting has broad applications, from adding interesting effects to casually captured movies to assisting video production professionals. Matting with associated effects such as shadows and reflections has also attr…
Image MattingVideo MattingGenerative Omnimatte: Learning to Decompose Video into Layers
Given a video and a set of input object masks, an omnimatte method aims to decompose the video into semantically meaningful layers containing individual objects along with their associated effects, such as shadows and re…
Depth EstimationVideo InpaintingEasyOmnimatte: Taming Pretrained Inpainting Diffusion Models for End-to-End Video Layered Decomposition
Existing video omnimatte methods typically rely on slow, multi-stage, or inference-time optimization pipelines that fail to fully exploit powerful generative priors, producing suboptimal decompositions. Our key insight i…
Video Inpainting