Rethinking Video Segmentation with Masked Video Consistency: Did the Model Learn as Intended?
Video segmentation aims at partitioning video sequences into meaningful segments based on objects or regions of interest within frames. Current video segmentation models are often derived from image segmentation techniques, which struggle to cope with small-scale or class-imbalanced video datasets. This leads to inconsistent segmentation results across frames. To address these issues, we propose a training strategy Masked Video Consistency, which enhances spatial and temporal feature aggregation. MVC introduces a training strategy that randomly masks image patches, compelling the network to predict the entire semantic segmentation, thus improving contextual information integration. Additionally, we introduce Object Masked Attention (OMA) to optimize the cross-attention mechanism by reducing the impact of irrelevant queries, thereby enhancing temporal modeling capabilities. Our approach, integrated into the latest decoupled universal video segmentation framework, achieves state-of-the-art performance across five datasets for three video segmentation tasks, demonstrating significant improvements over previous methods without increasing model parameters.
Code (0)
등록된 구현이 없습니다.
Tasks
Image SegmentationSegmentationSemantic SegmentationVideo SegmentationVideo Semantic SegmentationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Semantic Segmentation on VSPW Dataset through Masked Video Consistency
Pixel-level Video Understanding requires effectively integrating three-dimensional data in both spatial and temporal dimensions to learn accurate and stable semantic information from continuous frames. However, existing …
Semantic SegmentationVideo UnderstandingMATIS: Masked-Attention Transformers for Surgical Instrument Segmentation
We propose Masked-Attention Transformers for Surgical Instrument Segmentation (MATIS), a two-stage, fully transformer-based method that leverages modern pixel-wise attention mechanisms for instrument segmentation. MATIS …
SegmentationMotion-Saliency Complementary Masked Modeling for Point Cloud Video Understanding
Point cloud video representation learning is crucial for 3D dynamic scene understanding. In this paper, we propose MoSaiC, a novel Motion-Saliency Complementary masked modeling framework for self-supervised point cloud v…
Representation LearningSemantic SegmentationScene UnderstandingAction SegmentationVideoMAC: Video Masked Autoencoders Meet ConvNets
Recently, the advancement of self-supervised learning techniques, like masked autoencoders (MAE), has greatly influenced visual representation learning for images and videos. Nevertheless, it is worth noting that the pre…
Pose TrackingRepresentation LearningSelf-Supervised LearningSemantic Segmentation+2Masked Video Distillation: Rethinking Masked Feature Modeling for Self-supervised Video Representation Learning
Benefiting from masked visual modeling, self-supervised video representation learning has achieved remarkable progress. However, existing methods focus on learning representations from scratch through reconstructing low-…
Action ClassificationAction RecognitionRepresentation LearningSelf-Supervised Action Recognition