Balancing Shared and Task-Specific Representations: A Hybrid Approach to Depth-Aware Video Panoptic Segmentation
In this work, we present Multiformer, a novel approach to depth-aware video panoptic segmentation (DVPS) based on the mask transformer paradigm. Our method learns object representations that are shared across segmentation, monocular depth estimation, and object tracking subtasks. In contrast to recent unified approaches that progressively refine a common object representation, we propose a hybrid method using task-specific branches within each decoder block, ultimately fusing them into a shared representation at the block interfaces. Extensive experiments on the Cityscapes-DVPS and SemKITTI-DVPS datasets demonstrate that Multiformer achieves state-of-the-art performance across all DVPS metrics, outperforming previous methods by substantial margins. With a ResNet-50 backbone, Multiformer surpasses the previous best result by 3.0 DVPQ points while also improving depth estimation accuracy. Using a Swin-B backbone, Multiformer further improves performance by 4.0 DVPQ points. Multiformer also provides valuable insights into the design of multi-task decoder architectures.
Code (0)
등록된 구현이 없습니다.
Tasks
DecoderDepth-aware Video Panoptic SegmentationDepth EstimationMonocular Depth EstimationObjectObject TrackingPanoptic SegmentationVideo Panoptic SegmentationSimilar Papers 제목 키워드 기반
Private-Shared Disentangled Multimodal VAE for Learning of Hybrid Latent Representations
Multi-modal generative models represent an important family of deep models, whose goal is to facilitate representation learning on data with multiple views or modalities. However, current deep multi-modal models focus on…
DisentanglementRepresentation LearningAdversarial Continual Learning
Continual learning aims to learn new tasks without forgetting previously learned ones. We hypothesize that representations learned to solve each task in a sequence have a shared structure while containing some task-speci…
Continual Learningimage-classificationImage ClassificationContextual Attention Modulation: Towards Efficient Multi-Task Adaptation in Large Language Models
Large Language Models (LLMs) possess remarkable generalization capabilities but struggle with multi-task adaptation, particularly in balancing knowledge retention with task-specific specialization. Conventional fine-tuni…
Question AnsweringGeneral KnowledgeLogical ReasoningCode GenerationLearning aligned EEG representations with subject-specific encoders
Cross-subject EEG decoding promises more training data, but it also exposes neural networks to strong inter-subject distribution shifts. We study whether task supervision and architecture alone can learn subject-aligned …
Eeg DecodingHeterogeneous Multi-task Learning with Expert Diversity
Predicting multiple heterogeneous biological and medical targets is a challenge for traditional deep learning models. In contrast to single-task learning, in which a separate model is trained for each target, multi-task …
DiversityMixture-of-ExpertsMulti-Task Learning