Knowledge-driven Subspace Fusion and Gradient Coordination for Multi-modal Learning

2024-06-20 · Yupei Zhang, Xiaofei Wang, Fangliangzi Meng, Jin Tang, Chao Li

Multi-modal learning plays a crucial role in cancer diagnosis and prognosis. Current deep learning based multi-modal approaches are often limited by their abilities to model the complex correlations between genomics and histology data, addressing the intrinsic complexity of tumour ecosystem where both tumour and microenvironment contribute to malignancy. We propose a biologically interpretative and robust multi-modal learning framework to efficiently integrate histology images and genomics by decomposing the feature subspace of histology images and genomics, reflecting distinct tumour and microenvironment features. To enhance cross-modal interactions, we design a knowledge-driven subspace fusion scheme, consisting of a cross-modal deformable attention module and a gene-guided consistency strategy. Additionally, in pursuit of dynamically optimizing the subspace knowledge, we further propose a novel gradient coordination learning strategy. Extensive experiments demonstrate the effectiveness of the proposed method, outperforming state-of-the-art techniques in three downstream tasks of glioma diagnosis, tumour grading, and survival analysis. Our code is available at https://github.com/helenypzhang/Subspace-Multimodal-Learning.

📄 PDF Abstract BibTeX arXiv:2406.13979

Code (1)

helenypzhang/subspace-multimodal-learning 공식 구현 pytorch

Tasks

PrognosisSurvival Analysis

Methods 이 논문이 사용한 방법론

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Attention 설명 없음

Deformable Attention Module Deformable Attention Module is an attention module used in the Deformable DETR architecture, which seeks to overcome one…

Disentangled Multi-modal Learning of Histology and Transcriptomics for Cancer Characterization

2025-08-22 · Yupei Zhang, Xiaofei Wang, Anran Liu, Lequan Yu 외 arxiv

Histopathology remains the gold standard for cancer diagnosis and prognosis. With the advent of transcriptome profiling, multi-modal learning combining transcriptomics with histology offers more comprehensive information…

Knowledge Distillation

SALT: Subspace Alignment as an Auxiliary Learning Task for Domain Adaptation

2019-06-11 · Kowshik Thopalli, Jayaraman J. Thiagarajan, Rushil Anirudh, Pavan Turaga

Unsupervised domain adaptation aims to transfer and adapt knowledge learned from a labeled source domain to an unlabeled target domain. Key components of unsupervised domain adaptation include: (a) maximizing performance…

Auxiliary Learning Domain Adaptation Form Unsupervised Domain Adaptation

Learning Topology-Driven Multi-Subspace Fusion for Grassmannian Deep Network

2025-11-09 · Xuan Yu, Tianyang Xu arxiv

Grassmannian manifold offers a powerful carrier for geometric representation learning by modelling high-dimensional data as low-dimensional subspaces. However, existing approaches predominantly rely on static single-subs…

Representation Learning 3D Action Recognition

Mixture-of-Experts with Gradient Conflict-Driven Subspace Topology Pruning for Emergent Modularity

2025-12-23 · Yuxing Gan, Ziyu Lei arxiv

Mixture-of-Experts (MoE) architectures achieve parameter efficiency through conditional computation, yet contemporary designs suffer from two fundamental limitations: structural parameter isolation that causes catastroph…

iGSP:Implicit Gradient Subspace Projection for Efficient Continual Learning of Vision-Language Models

2026-05-19 · Xuezhi Cui, Dongbo Zhou, Wang Guo, Zeyuan Wang 외 arxiv

Vision-Language Models require efficient adaptation to continually emerging downstream tasks. While Parameter-Efficient Fine-Tuning mitigates catastrophic forgetting, assigning isolated modules per task leads to paramete…

parameter-efficient fine-tuning Continual Learning