Contextual Guided Segmentation Framework for Semi-supervised Video Instance Segmentation
In this paper, we propose Contextual Guided Segmentation (CGS) framework for video instance segmentation in three passes. In the first pass, i.e., preview segmentation, we propose Instance Re-Identification Flow to estimate main properties of each instance (i.e., human/non-human, rigid/deformable, known/unknown category) by propagating its preview mask to other frames. In the second pass, i.e., contextual segmentation, we introduce multiple contextual segmentation schemes. For human instance, we develop skeleton-guided segmentation in a frame along with object flow to correct and refine the result across frames. For non-human instance, if the instance has a wide variation in appearance and belongs to known categories (which can be inferred from the initial mask), we adopt instance segmentation. If the non-human instance is nearly rigid, we train FCNs on synthesized images from the first frame of a video sequence. In the final pass, i.e., guided segmentation, we develop a novel fined-grained segmentation method on non-rectangular regions of interest (ROIs). The natural-shaped ROI is generated by applying guided attention from the neighbor frames of the current one to reduce the ambiguity in the segmentation of different overlapping instances. Forward mask propagation is followed by backward mask propagation to further restore missing instance fragments due to re-appeared instances, fast motion, occlusion, or heavy deformation. Finally, instances in each frame are merged based on their depth values, together with human and non-human object interaction and rare instance priority. Experiments conducted on the DAVIS Test-Challenge dataset demonstrate the effectiveness of our proposed framework. We achieved the 3rd consistently in the DAVIS Challenges 2017-2019 with 75.4%, 72.4%, and 78.4% in terms of global score, region similarity, and contour accuracy, respectively.
Code (0)
등록된 구현이 없습니다.
Tasks
Human-Object Interaction DetectionInstance SegmentationSegmentationSemantic SegmentationVideo Instance SegmentationSimilar Papers 제목 키워드 기반
Medical Instrument Segmentation in 3D US by Hybrid Constrained Semi-Supervised Learning
Medical instrument segmentation in 3D ultrasound is essential for image-guided intervention. However, to train a successful deep neural network for instrument segmentation, a large number of labeled images are required, …
SegmentationSaliency Guided Self-attention Network for Weakly and Semi-supervised Semantic Segmentation
Weakly supervised semantic segmentation (WSSS) using only image-level labels can greatly reduce the annotation cost and therefore has attracted considerable research interest. However, its performance is still inferior t…
SegmentationSemantic SegmentationSemi-Supervised Semantic SegmentationWeakly supervised Semantic Segmentation+1UCAD: Uncertainty-guided Contour-aware Displacement for semi-supervised medical image segmentation
Existing displacement strategies in semi-supervised segmentation only operate on rectangular regions, ignoring anatomical structures and resulting in boundary distortions and semantic inconsistency. To address these issu…
Semi-supervised Medical Image SegmentationSpiderMesh: Spatial-aware Demand-guided Recursive Meshing for RGB-T Semantic Segmentation
For semantic segmentation in urban scene understanding, RGB cameras alone often fail to capture a clear holistic topology in challenging lighting conditions. Thermal signal is an informative additional channel that can b…
Data AugmentationSegmentationSemantic SegmentationSemi-Supervised Semantic Segmentation+1Mind the Context: Attention-Guided Weak-to-Strong Consistency for Enhanced Semi-Supervised Medical Image Segmentation
Medical image segmentation is a pivotal step in diagnostic and therapeutic processes, relying on high-quality annotated data that is often challenging and costly to obtain. Semi-supervised learning offers a promising app…
DiagnosticImage SegmentationMedical Image SegmentationSegmentation+2