A Text-Image Fusion Method with Data Augmentation Capabilities for Referring Medical Image Segmentation
Deep learning relies heavily on data augmentation to mitigate limited data, especially in medical imaging. Recent multimodal learning integrates text and images for segmentation, known as referring or text-guided image segmentation. However, common augmentations like rotation and flipping disrupt spatial alignment between image and text, weakening performance. To address this, we propose an early fusion framework that combines text and visual features before augmentation, preserving spatial consistency. We also design a lightweight generator that projects text embeddings into visual space, bridging semantic gaps. Visualization of generated pseudo-images shows accurate region localization. Our method is evaluated on three medical imaging tasks and four segmentation frameworks, achieving state-of-the-art results. Code is publicly available on GitHub: https://github.com/11yxk/MedSeg_EarlyFusion.
Code (0)
등록된 구현이 없습니다.
Tasks
Medical Image SegmentationData AugmentationSimilar Papers 제목 키워드 기반
3D-VirtFusion: Synthetic 3D Data Augmentation through Generative Diffusion Models and Controllable Editing
Data augmentation plays a crucial role in deep learning, enhancing the generalization and robustness of learning-based models. Standard approaches involve simple transformations like rotations and flips for generating ex…
Data AugmentationDiversityFew-Shot LearningScene UnderstandingSemantic Augmentation in Images using Language
Deep Learning models are incredibly data-hungry and require very large labeled datasets for supervised learning. As a consequence, these models often suffer from overfitting, limiting their ability to generalize to real-…
Data AugmentationDeep LearningDomain GeneralizationTTIDA: Controllable Generative Data Augmentation via Text-to-Text and Text-to-Image Models
Data augmentation has been established as an efficacious approach to supplement useful information for low-resource datasets. Traditional augmentation techniques such as noise injection and image transformations have bee…
Data AugmentationDiversitydomain classificationImage CaptioningPrompt-Free Conditional Diffusion for Multi-object Image Augmentation
Diffusion models has underpinned much recent advances of dataset augmentation in various computer vision tasks. However, when involving generating multi-object images as real scenarios, most existing methods either rely …
DiversityDomain GeneralizationImage AugmentationReinforcement Learning from Diffusion Feedback: Q* for Image Search
Large vision-language models are steadily gaining personalization capabilities at the cost of fine-tuning or data augmentation. We present two models for image generation using model-agnostic learning that align semantic…
Data AugmentationDiversityImage GenerationImage Retrieval+3