paper-with-me

홈 › Papers

PS3: A Multimodal Transformer Integrating Pathology Reports with Histology Images and Biological Pathways for Cancer Survival Prediction

2025-09-24 · Manahil Raza, Ayesha Azam, Talha Qaiser, Nasir Rajpoot arxiv

Current multimodal fusion approaches in computational oncology primarily focus on integrating multi-gigapixel histology whole slide images (WSIs) with genomic or transcriptomic data, demonstrating improved survival prediction. We hypothesize that incorporating pathology reports can further enhance prognostic performance. Pathology reports, as essential components of clinical workflows, offer readily available complementary information by summarizing histopathological findings and integrating expert interpretations and clinical context. However, fusing these modalities poses challenges due to their heterogeneous nature. WSIs are high-dimensional, each containing several billion pixels, whereas pathology reports consist of concise text summaries of varying lengths, leading to potential modality imbalance. To address this, we propose a prototype-based approach to generate balanced representations, which are then integrated using a Transformer-based fusion model for survival prediction that we term PS3 (Predicting Survival from Three Modalities). Specifically, we present: (1) Diagnostic prototypes from pathology reports, leveraging self-attention to extract diagnostically relevant sections and standardize text representation; (2) Histological prototypes to compactly represent key morphological patterns in WSIs; and (3) Biological pathway prototypes to encode transcriptomic expressions, accurately capturing cellular functions. PS3, the three-modal transformer model, processes the resulting prototype-based multimodal tokens and models intra-modal and cross-modal interactions across pathology reports, WSIs and transcriptomic data. The proposed model outperforms state-of-the-art methods when evaluated against clinical, unimodal and multimodal baselines on six datasets from The Cancer Genome Atlas (TCGA). The code is available at: https://github.com/manahilr/PS3.

📄 PDF Abstract BibTeX arXiv:2509.20022

Code (0)

등록된 구현이 없습니다.

Similar Papers 제목 키워드 기반

A Generative Foundation Model for Multimodal Histopathology

2026-04-04 · Jinxi Xiang, Mingjie Li, Siyu Hou, Yijiang Chen 외 arxiv

Accurate diagnosis and treatment of complex diseases require integrating histological, molecular, and clinical data, yet in practice these modalities are often incomplete owing to tissue scarcity, assay cost, and workflo…

Data AugmentationImage Generation

PEaRL: Pathway-Enhanced Representation Learning for Gene and Pathway Expression Prediction from Histology

2025-10-03 · Sejuti Majumder, Saarthak Kapse, Moinak Bhattacharya, Xuan Xu 외 arxiv

Integrating histopathology with spatial transcriptomics (ST) provides a powerful opportunity to link tissue morphology with molecular function. Yet most existing multimodal approaches rely on a small set of highly variab…

Representation LearningContrastive Learning

Multimodal Oncology Agent for IDH1 Mutation Prediction in Low-Grade Glioma

2025-12-05 · Hafsa Akebli, Adam Shephard, Vincenzo Della Mea, Nasir Rajpoot arxiv

Low-grade gliomas frequently present IDH1 mutations that define clinically distinct subgroups with specific prognostic and therapeutic implications. This work introduces a Multimodal Oncology Agent (MOA) integrating a hi…

MGCT: Mutual-Guided Cross-Modality Transformer for Survival Outcome Prediction using Integrative Histopathology-Genomic Features

2023-11-20 · Mingxin Liu, Yunzan Liu, Hui Cui, Chunquan Li 외

The rapidly emerging field of deep learning-based computational pathology has shown promising results in utilizing whole slide images (WSIs) to objectively prognosticate cancer patients. However, most prognostic methods …

Prognosiswhole slide images

Momentum Memory for Knowledge Distillation in Computational Pathology

2026-02-24 · Yongxin Guo, Hao Lu, Onur C. Koyun, Zhengjie Zhu 외 arxiv

Multimodal learning that integrates genomics and histopathology has shown strong potential in cancer diagnosis, yet its clinical translation is hindered by the limited availability of paired histology-genomics data. Know…

Knowledge Distillation