Multimodal Sentiment Analysis using Hierarchical Fusion with Context Modeling

2018-06-16 · N. Majumder, D. Hazarika, A. Gelbukh, E. Cambria, S. Poria

Multimodal sentiment analysis is a very actively growing field of research. A promising area of opportunity in this field is to improve the multimodal fusion mechanism. We present a novel feature fusion strategy that proceeds in a hierarchical fashion, first fusing the modalities two in two and only then fusing all three modalities. On multimodal sentiment analysis of individual utterances, our strategy outperforms conventional concatenation of features by 1%, which amounts to 5% reduction in error rate. On utterance-level multimodal sentiment analysis of multi-utterance video clips, for which current state-of-the-art techniques incorporate contextual information from other utterances of the same clip, our hierarchical fusion gives up to 2.4% (almost 10% error rate reduction) over currently used concatenation. The implementation of our method is publicly available in the form of open-source code.

📄 PDF Abstract BibTeX arXiv:1806.06228

Code (1)

SenticNet/hfusion tf

Tasks

Multimodal Emotion Recognition Multimodal Sentiment Analysis Sentiment Analysis

Toward Robust Incomplete Multimodal Sentiment Analysis via Hierarchical Representation Learning

2024-11-05 · Mingcheng Li, Dingkang Yang, Yang Liu, Shunli Wang 외

Multimodal Sentiment Analysis (MSA) is an important research area that aims to understand and recognize human sentiment through multiple modalities. The complementary information provided by multimodal fusion promotes be…

Multimodal Sentiment Analysis Representation Learning Sentiment Analysis

DashFusion: Dual-stream Alignment with Hierarchical Bottleneck Fusion for Multimodal Sentiment Analysis

2025-12-05 · Yuhua Wen, Qifei Li, Yingying Zhou, Yingming Gao 외 arxiv

Multimodal sentiment analysis (MSA) integrates various modalities, such as text, image, and audio, to provide a more comprehensive understanding of sentiment. However, effective MSA is challenged by alignment and fusion …

Multimodal Sentiment Analysis Computational Efficiency Contrastive Learning

PGF-Net: A Progressive Gated-Fusion Framework for Efficient Multimodal Sentiment Analysis

2025-08-20 · Bin Wen, Tien-Ping Tan arxiv

We introduce PGF-Net (Progressive Gated-Fusion Network), a novel deep learning framework designed for efficient and interpretable multimodal sentiment analysis. Our framework incorporates three primary innovations. First…

parameter-efficient fine-tuning Multimodal Sentiment Analysis Computational Efficiency

ConKI: Contrastive Knowledge Injection for Multimodal Sentiment Analysis

2023-06-27 · Yakun Yu, Mingjun Zhao, Shi-ang Qi, Feiran Sun 외

Multimodal Sentiment Analysis leverages multimodal signals to detect the sentiment of a speaker. Previous approaches concentrate on performing multimodal fusion and representation learning based on general knowledge obta…

Contrastive Learning General Knowledge Multimodal Sentiment Analysis Representation Learning+1

A Unified Framework for Emotion Recognition and Sentiment Analysis via Expert-Guided Multimodal Fusion with Large Language Models

2026-01-12 · Jiaqi Qiao, Xiujuan Xu, Xinran Li, Yu Liu arxiv

Multimodal emotion understanding requires effective integration of text, audio, and visual modalities for both discrete emotion recognition and continuous sentiment analysis. We present EGMF, a unified framework combinin…

Computational Efficiency Emotion Recognition Sentiment Analysis