paper-with-me

홈 › Papers

MUST: Modality-Specific Representation-Aware Transformer for Diffusion-Enhanced Survival Prediction with Missing Modality

2026-03-27 · Kyungwon Kim, Dosik Hwang arxiv

Accurate survival prediction from multimodal medical data is essential for precision oncology, yet clinical deployment faces a persistent challenge: modalities are frequently incomplete due to cost constraints, technical limitations, or retrospective data availability. While recent methods attempt to address missing modalities through feature alignment or joint distribution learning, they fundamentally lack explicit modeling of the unique contributions of each modality as opposed to the information derivable from other modalities. We propose MUST (Modality-Specific representation-aware Transformer), a novel framework that explicitly decomposes each modality's representation into modality-specific and cross-modal contextualized components through algebraic constraints in a learned low-rank shared subspace. This decomposition enables precise identification of what information is lost when a modality is absent. For the truly modality-specific information that cannot be inferred from available modalities, we employ conditional latent diffusion models to generate high-quality representations conditioned on recovered shared information and learned structural priors. Extensive experiments on five TCGA cancer datasets demonstrate that MUST achieves state-of-the-art performance with complete data while maintaining robust predictions in both missing pathology and missing genomics conditions, with clinically acceptable inference latency.

📄 PDF Abstract BibTeX arXiv:2603.26071

Code (0)

등록된 구현이 없습니다.

Similar Papers 제목 키워드 기반

MTNet: Learning modality-aware representation with transformer for RGBT tracking

2025-08-24 · Ruichao Hou, Boyue Xu, Tongwei Ren, Gangshan Wu arxiv

The ability to learn robust multi-modality representation has played a critical role in the development of RGBT tracking. However, the regular fusion paradigm and the invariable tracking template remain restrictive to th…

NestedFormer: Nested Modality-Aware Transformer for Brain Tumor Segmentation

2022-08-31 · Zhaohu Xing, Lequan Yu, Liang Wan, Tong Han 외

Multi-modal MR imaging is routinely used in clinical practice to diagnose and investigate brain tumors by providing rich complementary information. Previous multi-modal MRI segmentation methods usually perform modal fusi…

Brain Tumor SegmentationDecoderMRI segmentationSegmentation+1

SwinNet: Swin Transformer drives edge-aware RGB-D and RGB-T salient object detection

2022-04-12 · Zhengyi Liu, Yacheng Tan, Qian He, Yun Xiao

Convolutional neural networks (CNNs) are good at extracting contexture features within certain receptive fields, while transformers can model the global long-range dependency features. By absorbing the advantage of trans…

Decoderobject-detectionObject DetectionRGB-T Salient Object Detection+1

Similarity Guided Multimodal Fusion Transformer for Semantic Location Prediction in Social Media

2024-05-09 · Zhizhen Zhang, Ning Wang, Haojie Li, Zhihui Wang

Semantic location prediction aims to derive meaningful location insights from multimodal social media posts, offering a more contextual understanding of daily activities than using GPS coordinates. This task faces signif…

Language Modelling

Enhancing Visible-Infrared Person Re-identification with Modality- and Instance-aware Visual Prompt Learning

2024-06-18 · Ruiqi Wu, Bingliang Jiao, Wenxuan Wang, Meng Liu 외

The Visible-Infrared Person Re-identification (VI ReID) aims to match visible and infrared images of the same pedestrians across non-overlapped camera views. These two input modalities contain both invariant information,…

Person Re-IdentificationPrompt Learning