Mirror U-Net: Marrying Multimodal Fission with Multi-task Learning for Semantic Segmentation in Medical Imaging
Positron Emission Tomography (PET) and Computer Tomography (CT) are routinely used together to detect tumors. PET/CT segmentation models can automate tumor delineation, however, current multimodal models do not fully exploit the complementary information in each modality, as they either concatenate PET and CT data or fuse them at the decision level. To combat this, we propose Mirror U-Net, which replaces traditional fusion methods with multimodal fission by factorizing the multimodal representation into modality-specific branches and an auxiliary multimodal decoder. At these branches, Mirror U-Net assigns a task tailored to each modality to reinforce unimodal features while preserving multimodal features in the shared representation. In contrast to previous methods that use either fission or multi-task learning, Mirror U-Net combines both paradigms in a unified framework. We explore various task combinations and examine which parameters to share in the model. We evaluate Mirror U-Net on the AutoPET PET/CT and on the multimodal MSD BrainTumor datasets, demonstrating its effectiveness in multimodal segmentation and achieving state-of-the-art performance on both datasets. Our code will be made publicly available.
Code (0)
등록된 구현이 없습니다.
Tasks
DecoderMulti-Task LearningSemantic SegmentationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Supervised Multi-Modal Fission Learning
Learning from multimodal datasets can leverage complementary information and improve performance in prediction tasks. A commonly used strategy to account for feature correlations in high-dimensional datasets is the laten…
An automatic method for segmentation of fission tracks in epidote crystal photomicrographs
Manual identification of fission tracks has practical problems, such as variation due to observer-observation efficiency. An automatic processing method that could identify fission tracks in a photomicrograph could solve…
Inverse Design of Potential Singlet Fission Molecules using a Transfer Learning Based Approach
Singlet fission has emerged as one of the most exciting phenomena known to improve the efficiencies of different types of solar cells and has found uses in diverse optoelectronic applications. The range of available sing…
Transfer LearningModalityMirror: Improving Audio Classification in Modality Heterogeneity Federated Learning with Multimodal Distillation
Multimodal Federated Learning frequently encounters challenges of client modality heterogeneity, leading to undesired performances for secondary modality in multimodal learning. It is particularly prevalent in audiovisua…
Audio ClassificationFederated LearningKnowledge DistillationA self-adaptive and robust fission clustering algorithm via heat diffusion and maximal turning angle
Cluster analysis, which focuses on the grouping and categorization of similar elements, is widely used in various fields of research. A novel and fast clustering algorithm, fission clustering algorithm, is proposed in re…
Clustering