FreeSeg: Free Mask from Interpretable Contrastive Language-Image Pretraining for Semantic Segmentation
Fully supervised semantic segmentation learns from dense masks, which requires heavy annotation cost for closed set. In this paper, we use natural language as supervision without any pixel-level annotation for open world segmentation. We call the proposed framework as FreeSeg, where the mask is freely available from raw feature map of pretraining model. Compared with zero-shot or openset segmentation, FreeSeg doesn't require any annotated masks, and it widely predicts categories beyond class-agnostic unsupervised segmentation. Specifically, FreeSeg obtains free mask from Image-Text Similarity Map (ITSM) of Interpretable Contrastive Language-Image Pretraining (ICLIP). And our core improvements are the smoothed min pooling for dense ICLIP, with the partial label and pixel strategies for segmentation. Furthermore, FreeSeg is very straight forward without complex design like grouping, clustering or retrieval. Besides the simplicity, the performances of FreeSeg surpass previous state-of-the-art at large margins, e.g. 13.4% higher at mIoU on VOC dataset in the same settings.
Code (0)
등록된 구현이 없습니다.
Tasks
RetrievalSegmentationSemantic Segmentationtext similarityZero-Shot Semantic SegmentationSimilar Papers 제목 키워드 기반
FreeSeg-Diff: Training-Free Open-Vocabulary Segmentation with Diffusion Models
Foundation models have exhibited unprecedented capabilities in tackling many domains and tasks. Models such as CLIP are currently widely used to bridge cross-modal representations, and text-to-image diffusion models are …
Image GenerationImage SegmentationSegmentationSemantic Segmentation+1FreeSeg: Unified, Universal and Open-Vocabulary Image Segmentation
Recently, open-vocabulary learning has emerged to accomplish segmentation for arbitrary categories of text-based descriptions, which popularizes the segmentation system to more general-purpose application scenarios. Howe…
Image SegmentationInstance SegmentationOpen Vocabulary Panoptic SegmentationPanoptic Segmentation+3Learning with Free Object Segments for Long-Tailed Instance Segmentation
One fundamental challenge in building an instance segmentation model for a large number of classes in complex scenes is the lack of training examples, especially for rare objects. In this paper, we explore the possibilit…
Instance SegmentationObjectSemantic SegmentationPeek-a-Boo Reasoning: Contrastive Region Masking in MLLMs
We introduce Contrastive Region Masking (CRM), a training free diagnostic that reveals how multimodal large language models (MLLMs) depend on specific visual regions at each step of chain-of-thought (CoT) reasoning. Unli…
ARC-CT: Anatomy-Routed Contrastive Vision-Language Learning for 3D Chest CT
Contrastive vision-language learning uses paired chest CT volumes and radiology reports to learn abnormality classifiers without manually annotated labels. However, two characteristics of chest CT challenge conventional …
Contrastive Learning