Video-based Cross-modal Auxiliary Network for Multimodal Sentiment Analysis
Multimodal sentiment analysis has a wide range of applications due to its information complementarity in multimodal interactions. Previous works focus more on investigating efficient joint representations, but they rarely consider the insufficient unimodal features extraction and data redundancy of multimodal fusion. In this paper, a Video-based Cross-modal Auxiliary Network (VCAN) is proposed, which is comprised of an audio features map module and a cross-modal selection module. The first module is designed to substantially increase feature diversity in audio feature extraction, aiming to improve classification accuracy by providing more comprehensive acoustic representations. To empower the model to handle redundant visual features, the second module is addressed to efficiently filter the redundant visual frames during integrating audiovisual data. Moreover, a classifier group consisting of several image classification networks is introduced to predict sentiment polarities and emotion categories. Extensive experimental results on RAVDESS, CMU-MOSI, and CMU-MOSEI benchmarks indicate that VCAN is significantly superior to the state-of-the-art methods for improving the classification accuracy of multimodal sentiment analysis.
Code (1)
Tasks
ClassificationDiversityimage-classificationImage ClassificationMultimodal Sentiment AnalysisSentiment AnalysisSimilar Papers 제목 키워드 기반
OmniMER: Auxiliary-Enhanced LLM Adaptation for Indonesian Multimodal Emotion Recognition
Indonesian, spoken by over 200 million people, remains underserved in multimodal emotion recognition research despite its dominant presence on Southeast Asian social media platforms. We introduce IndoMER, the first multi…
Multimodal Emotion RecognitionKeyword ExtractionA Multimodal Sentiment Dataset for Video Recommendation
Recently, multimodal sentiment analysis has seen remarkable advance and a lot of datasets are proposed for its development. In general, current multimodal sentiment analysis datasets usually follow the traditional system…
Multimodal Sentiment AnalysisSentiment AnalysisVideo UnderstandingGated Mechanism for Attention Based Multimodal Sentiment Analysis
Multimodal sentiment analysis has recently gained popularity because of its relevance to social media posts, customer service calls and video blogs. In this paper, we address three aspects of multimodal sentiment analysi…
Multimodal Sentiment AnalysisSentiment AnalysisAdversarial Multimodal Domain Transfer for Video-Level Sentiment Analysis
Video-level sentiment analysis is a challenging task and requires systems to obtain discriminative multimodal representations that can capture difference in sentiments across various modalities. However, due to diverse …
Multimodal Sentiment AnalysisSentiment AnalysisA Novel Context-Aware Multimodal Framework for Persian Sentiment Analysis
Most recent works on sentiment analysis have exploited the text modality. However, millions of hours of video recordings posted on social media platforms everyday hold vital unstructured information that can be exploited…
Multimodal Sentiment AnalysisPersian Sentiment AnalysisSentiment Analysis