U4D: Unsupervised 4D Dynamic Scene Understanding
We introduce the first approach to solve the challenging problem of unsupervised 4D visual scene understanding for complex dynamic scenes with multiple interacting people from multi-view video. Our approach simultaneously estimates a detailed model that includes a per-pixel semantically and temporally coherent reconstruction, together with instance-level segmentation exploiting photo-consistency, semantic and motion information. We further leverage recent advances in 3D pose estimation to constrain the joint semantic instance segmentation and 4D temporally coherent reconstruction. This enables per person semantic instance segmentation of multiple interacting people in complex dynamic scenes. Extensive evaluation of the joint visual scene understanding framework against state-of-the-art methods on challenging indoor and outdoor sequences demonstrates a significant (approx 40%) improvement in semantic segmentation, reconstruction and scene flow accuracy.
Code (0)
등록된 구현이 없습니다.
Tasks
3D Pose EstimationInstance SegmentationPose EstimationScene UnderstandingSegmentationSemantic SegmentationSimilar Papers 제목 키워드 기반
4DContrast: Contrastive Learning with Dynamic Correspondences for 3D Scene Understanding
We present a new approach to instill 4D dynamic object priors into learned 3D representations by unsupervised pre-training. We observe that dynamic movement of an object through an environment provides important cues abo…
3D Instance Segmentation3D Semantic SegmentationContrastive LearningData Augmentation+9Feed-Forward SceneDINO for Unsupervised Semantic Scene Completion
Semantic scene completion (SSC) aims to infer both the 3D geometry and semantics of a scene from single images. In contrast to prior work on SSC that heavily relies on expensive ground-truth annotations, we approach SSC …
3D geometryDomain GeneralizationRepresentation LearningScene UnderstandingSeaDSC: A video-based unsupervised method for dynamic scene change detection in unmanned surface vehicles
Recently, there has been an upsurge in the research on maritime vision, where a lot of works are influenced by the application of computer vision for Unmanned Surface Vehicles (USVs). Various sensor modalities such as ca…
Change DetectionMotion Planningobject-detectionObject Detection+3Scene-Centric Unsupervised Panoptic Segmentation
Unsupervised panoptic segmentation aims to partition an image into semantically meaningful regions and distinct object instances without training on manually annotated data. In contrast to prior work on unsupervised pano…
Instance SegmentationPanoptic SegmentationPseudo LabelScene Understanding+5Dynamic Scene Understanding through Object-Centric Voxelization and Neural Rendering
Learning object-centric representations from unsupervised videos is challenging. Unlike most previous approaches that focus on decomposing 2D images, we present a 3D generative model named DynaVol-S for dynamic scenes th…
Inverse RenderingNeRFNeural RenderingNovel View Synthesis+3