ViP-DeepLab: Learning Visual Perception with Depth-aware Video Panoptic Segmentation
In this paper, we present ViP-DeepLab, a unified model attempting to tackle the long-standing and challenging inverse projection problem in vision, which we model as restoring the point clouds from perspective image sequences while providing each point with instance-level semantic interpretations. Solving this problem requires the vision models to predict the spatial location, semantic class, and temporally consistent instance label for each 3D point. ViP-DeepLab approaches it by jointly performing monocular depth estimation and video panoptic segmentation. We name this joint task as Depth-aware Video Panoptic Segmentation, and propose a new evaluation metric along with two derived datasets for it, which will be made available to the public. On the individual sub-tasks, ViP-DeepLab also achieves state-of-the-art results, outperforming previous methods by 5.1% VPQ on Cityscapes-VPS, ranking 1st on the KITTI monocular depth estimation benchmark, and 1st on KITTI MOTS pedestrian. The datasets and the evaluation codes are made publicly available.
Code (2)
Tasks
Depth-aware Video Panoptic SegmentationDepth EstimationMonocular Depth EstimationPanoptic SegmentationSegmentationVideo Panoptic SegmentationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Audio-Visual Segmentation via Depth-Guided Collaborative Modeling
Audio-Visual Segmentation (AVS) is a fundamental task in multimodal perception that performs pixel-level segmentation of sounding objects in videos by leveraging both visual and audio cues. It has broad applications in v…
Autonomous DrivingDriveCtrl: Conditioned Sim-to-Real Driving Video Generation
Large-scale labelled driving video data is essential for training autonomous driving systems. Although simulation offers scalable and fully annotated data, the domain gap between synthetic and real-world driving videos s…
Autonomous DrivingVideo GenerationMixed Reality using Illumination-aware Gradient Mixing in Surgical Telepresence: Enhanced Multi-layer Visualization
Background and aim: Surgical telepresence using augmented perception has been applied, but mixed reality is still being researched and is only theoretical. The aim of this work is to propose a solution to improve the vis…
Mixed RealityVideoCuRL: Video Curriculum Reinforcement Learning with Orthogonal Difficulty Decomposition
Reinforcement Learning (RL) is crucial for empowering VideoLLMs with complex spatiotemporal reasoning. However, current RL paradigms predominantly rely on random data shuffling or naive curriculum strategies based on sca…
Reinforcement LearningEgoDTM: Towards 3D-Aware Egocentric Video-Language Pretraining
Egocentric video-language pretraining has significantly advanced video representation learning. Humans perceive and interact with a fully 3D world, developing spatial awareness that extends beyond text-based understandin…
Contrastive LearningDecoderDepth EstimationRepresentation Learning