Learning Language-guided Adaptive Hyper-modality Representation for Multimodal Sentiment Analysis

2023-10-09 · Haoyu Zhang, Yu Wang, Guanghao Yin, Kejun Liu, Yuanyuan Liu, Tianshu Yu

Though Multimodal Sentiment Analysis (MSA) proves effective by utilizing rich information from multiple sources (e.g., language, video, and audio), the potential sentiment-irrelevant and conflicting information across modalities may hinder the performance from being further improved. To alleviate this, we present Adaptive Language-guided Multimodal Transformer (ALMT), which incorporates an Adaptive Hyper-modality Learning (AHL) module to learn an irrelevance/conflict-suppressing representation from visual and audio features under the guidance of language features at different scales. With the obtained hyper-modality representation, the model can obtain a complementary and joint representation through multimodal fusion for effective MSA. In practice, ALMT achieves state-of-the-art performance on several popular datasets (e.g., MOSI, MOSEI and CH-SIMS) and an abundance of ablation demonstrates the validity and necessity of our irrelevance/conflict suppression mechanism.

📄 PDF Abstract BibTeX arXiv:2310.05804

Code (1)

Haoyu-ha/ALMT 공식 구현 pytorch

Tasks

Multimodal Sentiment Analysis Sentiment Analysis

Methods 이 논문이 사용한 방법론

Multi-Head Attention 설명 없음

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Position-Wise Feed-Forward Layer 설명 없음

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

Adam 설명 없음

Dual Modality Prompted Diffusion Priors for Zero Shot Hyperspectral Pansharpening

2026-08-12 · Pengwei Xie, Fei Zhu, Jiajun Li, Xiangyuan Liu 외 arxiv

Hyperspectral pansharpening aims to reconstruct a high resolution hyperspectral (HRHS) image from a panchromatic (PAN) image and a low resolution hyperspectral (LRHS) image while preserving both spatial details and spect…

ModalImmune: Immunity Driven Unlearning via Self Destructive Training

2026-02-18 · Rong Fu, WeiZhi Tang, Ziming Wang, Jia Yee Tan 외 arxiv

Multimodal systems are vulnerable to partial or complete loss of input channels at deployment, which undermines reliability in real-world settings. This paper presents ModalImmune, a training framework that enforces moda…

Hyperbolic Distillation: Geometry-Guided Cross-Modal Transfer for Robust 3D Object Detection

2026-05-11 · Kanglin Ning, Wenrui Li, Houde Quan, Qifan Li 외 arxiv

Cross-modal knowledge distillation has emerged as an effective strategy for integrating point cloud and image features in 3D perception tasks. However, the modality heterogeneity, spatial misalignment, and the representa…

Robust 3D Object Detection Knowledge Distillation

Adaptive Hyper-graph Aggregation for Modality-Agnostic Federated Learning

2024-01-01 · CVPR 2024 1 · Fan Qi, Shuai Li

In Federated Learning (FL) the issue of statistical data heterogeneity has been a significant challenge to the field's ongoing development. This problem is further exacerbated when clients' data vary in modalities. I…

Federated Learning Graph Learning

VT-CLIP: Enhancing Vision-Language Models with Visual-guided Texts

2021-12-04 · Longtian Qiu, Renrui Zhang, Ziyu Guo, Ziyao Zeng 외

Contrastive Language-Image Pre-training (CLIP) has drawn increasing attention recently for its transferable visual representation learning. However, due to the semantic gap within datasets, CLIP's pre-trained image-text …

Language Modelling Representation Learning Zero-Shot Learning