Learning Monocular Depth in Dynamic Scenes via Instance-Aware Projection Consistency
We present an end-to-end joint training framework that explicitly models 6-DoF motion of multiple dynamic objects, ego-motion and depth in a monocular camera setup without supervision. Our technical contributions are three-fold. First, we highlight the fundamental difference between inverse and forward projection while modeling the individual motion of each rigid object, and propose a geometrically correct projection pipeline using a neural forward projection module. Second, we design a unified instance-aware photometric and geometric consistency loss that holistically imposes self-supervisory signals for every background and object region. Lastly, we introduce a general-purpose auto-annotation scheme using any off-the-shelf instance segmentation and optical flow models to produce video instance segmentation maps that will be utilized as input to our training pipeline. These proposed elements are validated in a detailed ablation study. Through extensive experiments conducted on the KITTI and Cityscapes dataset, our framework is shown to outperform the state-of-the-art depth and motion estimation methods. Our code, dataset, and models are available at https://github.com/SeokjuLee/Insta-DM .
Code (1)
Tasks
Instance SegmentationMonocular Depth EstimationMotion EstimationOptical Flow EstimationSemantic SegmentationUnsupervised Monocular Depth EstimationVideo Instance SegmentationSimilar Papers 제목 키워드 기반
Instance-aware multi-object self-supervision for monocular depth prediction
This paper proposes a self-supervised monocular image-to-depth prediction framework that is trained with an end-to-end photometric loss that handles not only 6-DOF camera motion but also 6-DOF moving object instances. Se…
Depth EstimationDepth PredictionObjectPose Estimation+1MoDGS: Dynamic Gaussian Splatting from Casually-captured Monocular Videos with Depth Priors
In this paper, we propose MoDGS, a new pipeline to render novel views of dy namic scenes from a casually captured monocular video. Previous monocular dynamic NeRF or Gaussian Splatting methods strongly rely on the rapid …
Depth EstimationNeRFEnhanced Scale-aware Depth Estimation for Monocular Endoscopic Scenes with Geometric Modeling
Scale-aware monocular depth estimation poses a significant challenge in computer-aided endoscopic navigation. However, existing depth estimation methods that do not consider the geometric priors struggle to learn the abs…
Depth EstimationMonocular Depth EstimationDO3D: Self-supervised Learning of Decomposed Object-aware 3D Motion and Depth from Monocular Videos
Although considerable advancements have been attained in self-supervised depth estimation from monocular videos, most existing methods often treat all objects in a video as static entities, which however violates the dyn…
Depth EstimationDisentanglementMotion DisentanglementMotion Estimation+3Manydepth2: Motion-Aware Self-Supervised Multi-Frame Monocular Depth Estimation in Dynamic Scenes
Despite advancements in self-supervised monocular depth estimation, challenges persist in dynamic scenarios due to the dependence on assumptions about a static world. In this paper, we present Manydepth2, to achieve prec…
Camera Pose EstimationComputational EfficiencyDepth EstimationMonocular Depth Estimation+1