Integrating Extra Modality Helps Segmentor Find Camouflaged Objects Well
Camouflaged Object Segmentation (COS) remains challenging because camouflaged objects exhibit only subtle visual differences from their backgrounds and single-modality RGB methods provide limited cues, leading researchers to explore multimodal data to improve segmentation accuracy. In this work, we presenet MultiCOS, a novel framework that effectively leverages diverse data modalities to improve segmentation performance. MultiCOS comprises two modules: Bi-space Fusion Segmentor (BFSer), which employs a state space and a latent space fusion mechanism to integrate cross-modal features within a shared representation and employs a fusion-feedback mechanism to refine context-specific features, and Cross-modal Knowledge Learner (CKLer), which leverages external multimodal datasets to generate pseudo-modal inputs and establish cross-modal semantic associations, transferring knowledge to COS models when real multimodal pairs are missing. When real multimodal COS data are unavailable, CKLer yields additional segmentation gains using only non-COS multimodal sources. Experiments on standard COS benchmarks show that BFSer outperforms existing multimodal baselines with both real and pseudo-modal data. Code will be released at \href{https://github.com/cnyvfang/MultiCOS}{GitHub}.
Code (2)
Tasks
Camouflaged Object SegmentationSegmentationSemantic SegmentationSimilar Papers 제목 키워드 기반
Towards Cross-modality Medical Image Segmentation with Online Mutual Knowledge Distillation
The success of deep convolutional neural networks is partially attributed to the massive amount of annotated training data. However, in practice, medical data annotations are usually expensive and time-consuming to be ob…
Cardiac SegmentationImage SegmentationKnowledge DistillationMedical Image Segmentation+2Translating and Segmenting Multimodal Medical Volumes with Cycle- and Shape-Consistency Generative Adversarial Network
Synthesized medical images have several important applications, e.g., as an intermedium in cross-modality image registration and as supplementary training samples to boost the generalization capability of a classifier. E…
Computed Tomography (CT)Generative Adversarial NetworkImage GenerationImage Registration+1Learning Robust Anymodal Segmentor with Unimodal and Cross-modal Distillation
Simultaneously using multimodal inputs from multiple sensors to train segmentors is intuitively advantageous but practically challenging. A key challenge is unimodal bias, where multimodal segmentors over rely on certain…
PAN: Projective Adversarial Network for Medical Image Segmentation
Adversarial learning has been proven to be effective for capturing long-range and high-level label consistencies in semantic segmentation. Unique to medical imaging, capturing 3D semantics in an effective yet computation…
Image SegmentationMedical Image SegmentationPancreas SegmentationSegmentation+1OmniSegmentor: A Flexible Multi-Modal Learning Framework for Semantic Segmentation
Recent research on representation learning has proved the merits of multi-modal clues for robust semantic segmentation. Nevertheless, a flexible pretrain-and-finetune pipeline for multiple visual modalities remains unexp…
Representation LearningSemantic Segmentation