INT: Instance-Specific Negative Mining for Task-Generic Promptable Segmentation
Task-generic promptable image segmentation aims to achieve segmentation of diverse samples under a single task description by utilizing only one task-generic prompt. Current methods leverage the generalization capabilities of Vision-Language Models (VLMs) to infer instance-specific prompts from these task-generic prompts in order to guide the segmentation process. However, when VLMs struggle to generalise to some image instances, predicting instance-specific prompts becomes poor. To solve this problem, we introduce \textbf{I}nstance-specific \textbf{N}egative Mining for \textbf{T}ask-Generic Promptable Segmentation (\textbf{INT}). The key idea of INT is to adaptively reduce the influence of irrelevant (negative) prior knowledge whilst to increase the use the most plausible prior knowledge, selected by negative mining with higher contrast, in order to optimise instance-specific prompts generation. Specifically, INT consists of two components: (1) instance-specific prompt generation, which progressively fliters out incorrect information in prompt generation; (2) semantic mask generation, which ensures each image instance segmentation matches correctly the semantics of the instance-specific prompts. INT is validated on six datasets, including camouflaged objects and medical images, demonstrating its effectiveness, robustness and scalability.
Code (0)
등록된 구현이 없습니다.
Tasks
Image SegmentationInstance SegmentationSegmentationSemantic SegmentationSimilar Papers 제목 키워드 기반
Trading Hard Negatives and True Negatives: A Debiased Contrastive Collaborative Filtering Approach
Collaborative filtering (CF), as a standard method for recommendation with implicit feedback, tackles a semi-supervised learning problem where most interaction data are unobserved. Such a nature makes existing approaches…
Collaborative FilteringAffinity Uncertainty-based Hard Negative Mining in Graph Contrastive Learning
Hard negative mining has shown effective in enhancing self-supervised contrastive learning (CL) on diverse data types, including graph CL (GCL). The existing hardness-aware CL methods typically treat negative instances t…
Contrastive LearningNode ClassificationTripletCo-mining: Self-Supervised Learning for Sparsely Annotated Object Detection
Object detectors usually achieve promising results with the supervision of complete instance annotations. However, their performance is far from satisfactory with sparse instance annotations. Most existing methods for sp…
MULTI-VIEW LEARNINGObjectobject-detectionObject Detection+2Stochastic Class-Based Hard Example Mining for Deep Metric Learning
Performance of deep metric learning depends heavily on the capability of mining hard negative examples during training. However, many metric learning algorithms often require intractable computational cost due to freque…
Image RetrievalMetric LearningRetrievalMemory Enhanced Embedding Learning for Cross-Modal Video-Text Retrieval
Cross-modal video-text retrieval, a challenging task in the field of vision and language, aims at retrieving corresponding instance giving sample from either modality. Existing approaches for this task all focus on how t…
RetrievalText RetrievalVideo-Text Retrieval