Towards Agnostic and Holistic Universal Image Segmentation with Bit Diffusion
This paper introduces a diffusion-based framework for universal image segmentation, making agnostic segmentation possible without depending on mask-based frameworks and instead predicting the full segmentation in a holistic manner. We present several key adaptations to diffusion models, which are important in this discrete setting. Notably, we show that a location-aware palette with our 2D gray code ordering improves performance. Adding a final tanh activation function is crucial for discrete data. On optimizing diffusion parameters, the sigmoid loss weighting consistently outperforms alternatives, regardless of the prediction type used, and we settle on x-prediction. While our current model does not yet surpass leading mask-based architectures, it narrows the performance gap and introduces unique capabilities, such as principled ambiguity modeling, that these models lack. All models were trained from scratch, and we believe that combining our proposed improvements with large-scale pretraining or promptable conditioning could lead to competitive models.
Code (0)
등록된 구현이 없습니다.
Tasks
Image SegmentationSimilar Papers 제목 키워드 기반
Uni-3D: A Universal Model for Panoptic 3D Scene Reconstruction
Performing holistic 3D scene understanding from a single-view observation, involving generating instance shapes and 3D scene segmentation, is a long-standing challenge. Prevailing works either focus only on geometry …
3D Scene ReconstructionImage SegmentationPanoptic SegmentationScene Parsing+4DFormer: Diffusion-guided Transformer for Universal Image Segmentation
This paper introduces an approach, named DFormer, for universal image segmentation. The proposed DFormer views universal image segmentation task as a denoising process using a diffusion model. DFormer first adds various …
DecoderDenoisingImage SegmentationInstance Segmentation+3HoliSDiP: Image Super-Resolution via Holistic Semantics and Diffusion Prior
Text-to-image diffusion models have emerged as powerful priors for real-world image super-resolution (Real-ISR). However, existing methods may produce unintended results due to noisy text prompts and their lack of spatia…
Image Super-ResolutionSegmentationSemantic SegmentationSuper-ResolutionUniversal Adversarial Perturbations Against Semantic Image Segmentation
While deep learning is remarkably successful on perceptual tasks, it was also shown to be vulnerable to adversarial perturbations of the input. These perturbations denote noise added to the input that was generated speci…
image-classificationImage ClassificationImage SegmentationSegmentation+1Towards Universal Dataset Distillation via Task-Driven Diffusion
Dataset distillation (DD) condenses key information from large-scale datasets into smaller synthetic datasets, reducing storage and computational costs for training networks. However, recent research has primarily fo…
Dataset Distillationimage-classificationImage ClassificationImage Generation