Self-supervised Semantic Segmentation Grounded in Visual Concepts
Unsupervised semantic segmentation requires assigning a label to every pixel without any human annotations. Despite recent advances in self-supervised representation learning for individual images, unsupervised semantic segmentation with pixel-level representations is still a challenging task and remains underexplored. In this work, we propose a self-supervised pixel representation learning method for semantic segmentation by using visual concepts (i.e., groups of pixels with semantic meanings, such as parts, objects, and scenes) extracted from images. To guide self-supervised learning, we leverage three types of relationships between pixels and concepts, including the relationships between pixels and local concepts, local and global concepts, as well as the co-occurrence of concepts. We evaluate the learned pixel embeddings and visual concepts on three datasets, including PASCAL VOC 2012, COCO 2017, and DAVIS 2017. Our results show that the proposed method gains consistent and substantial improvements over recent unsupervised semantic segmentation approaches, and also demonstrate that visual concepts can reveal insights into image datasets.
Code (0)
등록된 구현이 없습니다.
Tasks
Representation LearningSegmentationSelf-Supervised LearningSemantic SegmentationUnsupervised Semantic SegmentationSimilar Papers 제목 키워드 기반
DatUS^2: Data-driven Unsupervised Semantic Segmentation with Pre-trained Self-supervised Vision Transformer
Successive proposals of several self-supervised training schemes continue to emerge, taking one step closer to developing a universal foundation model. In this process, the unsupervised downstream tasks are recognized as…
SegmentationSemantic SegmentationUnsupervised Semantic SegmentationUnderstanding Self-Supervised Features for Learning Unsupervised Instance Segmentation
Self-supervised learning (SSL) can be used to solve complex visual tasks without human labels. Self-supervised representations encode useful semantic information about images, and as a result, they have already been used…
Instance SegmentationSegmentationSelf-Supervised LearningSemantic Segmentation+3Word Discovery in Visually Grounded, Self-Supervised Speech Models
We present a method for visually-grounded spoken term discovery. After training either a HuBERT or wav2vec2.0 model to associate spoken captions with natural images, we show that powerful word segmentation and clustering…
ClusteringSegmentationVisual GroundingSelf-Supervised Difference Detection for Weakly-Supervised Semantic Segmentation
To minimize the annotation costs associated with the training of semantic segmentation models, researchers have extensively investigated weakly-supervised segmentation approaches. In the current weakly-supervised segment…
SegmentationSemantic SegmentationWeakly supervised segmentationWeakly supervised Semantic Segmentation+1Syllable Discovery and Cross-Lingual Generalization in a Visually Grounded, Self-Supervised Speech Model
In this paper, we show that representations capturing syllabic units emerge when training a self-supervised speech model with a visually-grounded training objective. We demonstrate that a nearly identical model architect…
Language ModelingLanguage ModellingMasked Language ModelingSegmentation+2