Analyzing Modality Robustness in Multimodal Sentiment Analysis
Building robust multimodal models are crucial for achieving reliable deployment in the wild. Despite its importance, less attention has been paid to identifying and improving the robustness of Multimodal Sentiment Analysis (MSA) models. In this work, we hope to address that by (i) Proposing simple diagnostic checks for modality robustness in a trained multimodal model. Using these checks, we find MSA models to be highly sensitive to a single modality, which creates issues in their robustness; (ii) We analyze well-known robust training strategies to alleviate the issues. Critically, we observe that robustness can be achieved without compromising on the original performance. We hope our extensive study-performed across five models and two benchmark datasets-and proposed procedures would make robustness an integral component in MSA research. Our diagnostic checks and robust training solutions are simple to implement and available at https://github. com/declare-lab/MSA-Robustness.
Code (1)
Tasks
DiagnosticMultimodal Sentiment AnalysisSentiment AnalysisSimilar Papers 제목 키워드 기반
Analyzing Modality Robustness in Multimodal Sentiment Analysis
Building robust multimodal models are crucial to achieving reliable deployment in the wild. Despite its importance, less attention has been paid to identifying and improving the robustness of Multimodal Sentiment Analysi…
DiagnosticMultimodal Sentiment AnalysisSentiment AnalysisModality-Invariant Bidirectional Temporal Representation Distillation Network for Missing Multimodal Sentiment Analysis
Multimodal Sentiment Analysis (MSA) integrates diverse modalities(text, audio, and video) to comprehensively analyze and understand individuals' emotional states. However, the real-world prevalence of incomplete data pos…
Multimodal Sentiment AnalysisRepresentation LearningSentiment AnalysisDravidianMultiModality: A Dataset for Multi-modal Sentiment Analysis in Tamil and Malayalam
Human communication is inherently multimodal and asynchronous. Analyzing human emotions and sentiment is an emerging field of artificial intelligence. We are witnessing an increasing amount of multimodal content in local…
Multimodal Sentiment AnalysisSentiment AnalysisBACN: Bi-direction Attention Capsule-based Network for Multimodal Sentiment Analysis
Capsule-based network has currently identified its effectiveness in analyzing the heterogeneity issue of multimodal sentiment analysis. However, existing manners could only exploit the spatial relation between represent…
Multimodal Sentiment AnalysisSentiment AnalysisKnowledge-Guided Dynamic Modality Attention Fusion Framework for Multimodal Sentiment Analysis
Multimodal Sentiment Analysis (MSA) utilizes multimodal data to infer the users' sentiment. Previous methods focus on equally treating the contribution of each modality or statically using text as the dominant modality t…
Multimodal Sentiment AnalysisSentiment Analysis