SHAP-CAT: A interpretable multi-modal framework enhancing WSI classification via virtual staining and shapley-value-based multimodal fusion

2024-10-02 · Jun Wang, Yu Mao, Nan Guan, Chun Jason Xue

The multimodal model has demonstrated promise in histopathology. However, most multimodal models are based on H\&E and genomics, adopting increasingly complex yet black-box designs. In our paper, we propose a novel interpretable multimodal framework named SHAP-CAT, which uses a Shapley-value-based dimension reduction technique for effective multimodal fusion. Starting with two paired modalities -- H\&E and IHC images, we employ virtual staining techniques to enhance limited input data by generating a new clinical-related modality. Lightweight bag-level representations are extracted from image modalities and a Shapley-value-based mechanism is used for dimension reduction. For each dimension of the bag-level representation, attribution values are calculated to indicate how changes in the specific dimensions of the input affect the model output. In this way, we select a few top important dimensions of bag-level representation for each image modality to late fusion. Our experimental results demonstrate that the proposed SHAP-CAT framework incorporating synthetic modalities significantly enhances model performance, yielding a 5\% increase in accuracy for the BCI, an 8\% increase for IHC4BC-ER, and an 11\% increase for the IHC4BC-PR dataset.

📄 PDF Abstract BibTeX arXiv:2410.01408

Code (0)

등록된 구현이 없습니다.

Tasks

Dimensionality ReductionVirtual Staining

Disentangled and Interpretable Multimodal Attention Fusion for Cancer Survival Prediction

2025-03-20 · Aniek Eijpe, Soufyan Lakbir, Melis Erdal Cesur, Sara P. Oliveira 외

To improve the prediction of cancer survival using whole-slide images and transcriptomics data, it is crucial to capture both modality-shared and modality-specific information. However, multimodal frameworks often entang…

Disentanglement Survival Prediction whole slide images

Uncertainty-Aware Multimodal Learning via Conformal Shapley Intervals

2026-01-30 · Mathew Chandy, Michael Johnson, Judong Shen, Devan V. Mehrotra 외 arxiv

Multimodal learning combines information from multiple data modalities to improve predictive performance. However, modalities often contribute unequally and in a data dependent way, making it unclear which data modalitie…

Predicting Mood Disorder Symptoms with Remotely Collected Videos Using an Interpretable Multimodal Dynamic Attention Fusion Network

2021-09-07 · Tathagata Banerjee, Matthew Kollada, Pablo Gersberg, Oscar Rodriguez 외

We developed a novel, interpretable multimodal classification method to identify symptoms of mood disorders viz. depression, anxiety and anhedonia using audio, video and text collected from a smartphone application. We u…

Unveiling Covert Toxicity in Multimodal Data via Toxicity Association Graphs: A Graph-Based Metric and Interpretable Detection Framework

2026-02-03 · Guanzong Wu, Zihao Zhu, Siwei Lyu, Baoyuan Wu arxiv

Detecting toxicity in multimodal data remains a significant challenge, as harmful meanings often lurk beneath seemingly benign individual modalities: only emerging when modalities are combined and semantic associations a…

Bridging the gap between Performance and Interpretability: An Explainable Disentangled Multimodal Framework for Cancer Survival Prediction

2026-03-02 · Aniek Eijpe, Soufyan Lakbir, Melis Erdal Cesur, Sara P. Oliveira 외 arxiv

While multimodal survival prediction models are increasingly more accurate, their complexity often reduces interpretability, limiting insight into how different data sources influence predictions. To address this, we int…