MaskDiffusion: Exploiting Pre-trained Diffusion Models for Semantic Segmentation
Semantic segmentation is essential in computer vision for various applications, yet traditional approaches face significant challenges, including the high cost of annotation and extensive training for supervised learning. Additionally, due to the limited predefined categories in supervised learning, models typically struggle with infrequent classes and are unable to predict novel classes. To address these limitations, we propose MaskDiffusion, an innovative approach that leverages pretrained frozen Stable Diffusion to achieve open-vocabulary semantic segmentation without the need for additional training or annotation, leading to improved performance compared to similar methods. We also demonstrate the superior performance of MaskDiffusion in handling open vocabularies, including fine-grained and proper noun-based categories, thus expanding the scope of segmentation applications. Overall, our MaskDiffusion shows significant qualitative and quantitative improvements in contrast to other comparable unsupervised segmentation methods, i.e. on the Potsdam dataset (+10.5 mIoU compared to GEM) and COCO-Stuff (+14.8 mIoU compared to DiffSeg). All code and data will be released at https://github.com/Valkyrja3607/MaskDiffusion.
Code (1)
Tasks
Open Vocabulary Semantic SegmentationOpen-Vocabulary Semantic SegmentationProper NounSegmentationSemantic SegmentationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
MaskDiffusion: Boosting Text-to-Image Consistency with Conditional Mask
Recent advancements in diffusion models have showcased their impressive capacity to generate visually striking images. Nevertheless, ensuring a close match between the generated image and the given prompt remains a persi…
Zero-Shot Video Semantic Segmentation based on Pre-Trained Diffusion Models
We introduce the first zero-shot approach for Video Semantic Segmentation (VSS) based on pre-trained diffusion models. A growing research direction attempts to employ diffusion models to perform downstream vision tasks b…
SegmentationSemantic correspondenceSemantic SegmentationVideo Semantic SegmentationVidSeg: Training-free Video Semantic Segmentation based on Diffusion Models
We introduce the first training-free approach for Video Semantic Segmentation (VSS) based on pre-trained diffusion models. A growing research direction attempts to employ diffusion models to perform downstream vision…
SegmentationSemantic SegmentationVideo Semantic SegmentationEmerDiff: Emerging Pixel-level Semantic Knowledge in Diffusion Models
Diffusion models have recently received increasing research attention for their remarkable transfer abilities in semantic segmentation tasks. However, generating fine-grained segmentation masks with diffusion models ofte…
SegmentationSemantic SegmentationPDDM: Pseudo Depth Diffusion Model for RGB-PD Semantic Segmentation Based in Complex Indoor Scenes
The integration of RGB and depth modalities significantly enhances the accuracy of segmenting complex indoor scenes, with depth data from RGB-D cameras playing a crucial role in this improvement. However, collecting an R…
Depth EstimationSegmentationSemantic Segmentation