Multi-Grained Cross-modal Alignment for Learning Open-vocabulary Semantic Segmentation from Text Supervision
Recently, learning open-vocabulary semantic segmentation from text supervision has achieved promising downstream performance. Nevertheless, current approaches encounter an alignment granularity gap owing to the absence of dense annotations, wherein they learn coarse image/region-text alignment during training yet perform group/pixel-level predictions at inference. Such discrepancy leads to suboptimal learning efficiency and inferior zero-shot segmentation results. In this paper, we introduce a Multi-Grained Cross-modal Alignment (MGCA) framework, which explicitly learns pixel-level alignment along with object- and region-level alignment to bridge the granularity gap without any dense annotations. Specifically, MGCA ingeniously constructs pseudo multi-granular semantic correspondences upon image-text pairs and collaborates with hard sampling strategies to facilitate fine-grained cross-modal contrastive learning. Further, we point out the defects of existing group and pixel prediction units in downstream segmentation and develop an adaptive semantic unit which effectively mitigates their dilemmas including under- and over-segmentation. Training solely on CC3M, our method achieves significant advancements over state-of-the-art methods, demonstrating its effectiveness and efficiency.
Code (0)
등록된 구현이 없습니다.
Tasks
Contrastive Learningcross-modal alignmentOpen Vocabulary Semantic SegmentationOpen-Vocabulary Semantic SegmentationSegmentationSemantic SegmentationZero Shot SegmentationSimilar Papers 제목 키워드 기반
CMI-MTL: Cross-Mamba interaction based multi-task learning for medical visual question answering
Medical visual question answering (Med-VQA) is a crucial multimodal task in clinical decision support and telemedicine. Recent self-attention based methods struggle to effectively handle cross-modal semantic alignments b…
Visual Question AnsweringMulti-Task LearningMulti-modal Attribute Prompting for Vision-Language Models
Pre-trained Vision-Language Models (VLMs), like CLIP, exhibit strong generalization ability to downstream tasks but struggle in few-shot scenarios. Existing prompting techniques primarily focus on global text and image r…
Attributecross-modal alignmentDecomposed Vision-Language Alignment for Fine-Grained Open-Vocabulary Segmentation
Open-vocabulary segmentation models often struggle to generalize to unseen combinations of object categories and attributes, because fine-grained descriptions are typically encoded as holistic sentences that entangle mul…
Learning Relative Representations for Fine-Grained Multimodal Alignment with Limited Data
Multimodal pre-training demonstrates strong generalization performance, but this paradigm is often impractical in domains where paired data are scarce. A promising alternative is post-hoc multimodal alignment, which alig…
Cross-Modal RetrievalLinguistic-Aware Patch Slimming Framework for Fine-grained Cross-Modal Alignment
Cross-modal alignment aims to build a bridge connecting vision and language. It is an important multi-modal task that efficiently learns the semantic similarities between images and texts. Traditional fine-grained al…
cross-modal alignmentCross-Modal RetrievalImage RetrievalImage-to-Text Retrieval+4