Towards Multimodal Sentiment Analysis Debiasing via Bias Purification
Multimodal Sentiment Analysis (MSA) aims to understand human intentions by integrating emotion-related clues from diverse modalities, such as visual, language, and audio. Unfortunately, the current MSA task invariably suffers from unplanned dataset biases, particularly multimodal utterance-level label bias and word-level context bias. These harmful biases potentially mislead models to focus on statistical shortcuts and spurious correlations, causing severe performance bottlenecks. To alleviate these issues, we present a Multimodal Counterfactual Inference Sentiment (MCIS) analysis framework based on causality rather than conventional likelihood. Concretely, we first formulate a causal graph to discover harmful biases from already-trained vanilla models. In the inference phase, given a factual multimodal input, MCIS imagines two counterfactual scenarios to purify and mitigate these biases. Then, MCIS can make unbiased decisions from biased observations by comparing factual and counterfactual outcomes. We conduct extensive experiments on several standard MSA benchmarks. Qualitative and quantitative results show the effectiveness of the proposed framework.
Code (0)
등록된 구현이 없습니다.
Tasks
counterfactualCounterfactual InferenceMultimodal Sentiment AnalysisSentiment AnalysisMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
General Debiasing for Multimodal Sentiment Analysis
Existing work on Multimodal Sentiment Analysis (MSA) utilizes multimodal information for prediction yet unavoidably suffers from fitting the spurious correlations between multimodal features and sentiment labels. For exa…
Multimodal Sentiment AnalysisSentiment AnalysisBeyond Spurious Signals: Debiasing Multimodal Large Language Models via Counterfactual Inference and Adaptive Expert Routing
Multimodal Large Language Models (MLLMs) have shown substantial capabilities in integrating visual and textual information, yet frequently rely on spurious correlations, undermining their robustness and generalization in…
Multimodal ReasoningSentiment AnalysisSarcasm DetectionMultimodal Sentiment Analysis Based on Causal Reasoning
With the rapid development of multimedia, the shift from unimodal textual sentiment analysis to multimodal image-text sentiment analysis has obtained academic and industrial attention in recent years. However, multimodal…
Causal InferencecounterfactualCounterfactual InferenceMultimodal Sentiment Analysis+2Target-oriented Multimodal Sentiment Classification with Counterfactual-enhanced Debiasing
Target-oriented multimodal sentiment classification seeks to predict sentiment polarity for specific targets from image-text pairs. While existing works achieve competitive performance, they often over-rely on textual co…
Contrastive LearningData AugmentationDINER: Debiasing Aspect-based Sentiment Analysis with Multi-variable Causal Inference
Though notable progress has been made, neural-based aspect-based sentiment analysis (ABSA) models are prone to learn spurious correlations from annotation biases, resulting in poor robustness on adversarial data transfor…
Aspect-Based Sentiment AnalysisAspect-Based Sentiment Analysis (ABSA)Causal Inferencecounterfactual+2