Multimodal Sentiment Analysis with Missing Modality: A Knowledge-Transfer Approach
Multimodal sentiment analysis aims to identify the emotions expressed by individuals through visual, language, and acoustic cues. However, most of the existing research efforts assume that all modalities are available during both training and testing, making their algorithms susceptible to the missing modality scenario. In this paper, we propose a novel knowledge-transfer network to translate between different modalities to reconstruct the missing audio modalities. Moreover, we develop a cross-modality attention mechanism to retain the maximal information of the reconstructed and observed modalities for sentiment prediction. Extensive experiments on three publicly available datasets demonstrate significant improvements over baselines and achieve comparable results to the previous methods with complete multi-modality supervision.
Code (0)
등록된 구현이 없습니다.
Tasks
Multimodal Sentiment AnalysisSentiment AnalysisTransfer LearningSimilar Papers 제목 키워드 기반
Modality-Invariant Bidirectional Temporal Representation Distillation Network for Missing Multimodal Sentiment Analysis
Multimodal Sentiment Analysis (MSA) integrates diverse modalities(text, audio, and video) to comprehensively analyze and understand individuals' emotional states. However, the real-world prevalence of incomplete data pos…
Multimodal Sentiment AnalysisRepresentation LearningSentiment AnalysisFSRF: Factorization-guided Semantic Recovery for Incomplete Multimodal Sentiment Analysis
In recent years, Multimodal Sentiment Analysis (MSA) has become a research hotspot that aims to utilize multimodal data for human sentiment understanding. Previous MSA studies have mainly focused on performing interactio…
Multimodal Sentiment AnalysisRepresentation LearningMissing Modality meets Meta Sampling (M3S): An Efficient Universal Approach for Multimodal Sentiment Analysis with Missing Modality
Multimodal sentiment analysis (MSA) is an important way of observing mental activities with the help of data captured from multiple modalities. However, due to the recording or transmission error, some modalities may inc…
Meta-LearningMultimodal Sentiment AnalysisSentiment AnalysisCorrelation-Decoupled Knowledge Distillation for Multimodal Sentiment Analysis with Incomplete Modalities
Multimodal sentiment analysis (MSA) aims to understand human sentiment through multimodal data. Most MSA efforts are based on the assumption of modality completeness. However, in real-world applications, some practical f…
DisentanglementKnowledge DistillationMultimodal Sentiment AnalysisSentiment AnalysisControlling Decision Drift in Multimodal Sentiment Analysis with Missing Modalities
Multimodal sentiment analysis relies on textual, acoustic, and visual signals, yet real-world data often suffer from modality missing and quality imbalance. Existing methods generate features for modality missing from av…
Multimodal Sentiment Analysis