ScaleVLAD: Improving Multimodal Sentiment Analysis via Multi-Scale Fusion of Locally Descriptors
Fusion technique is a key research topic in multimodal sentiment analysis. The recent attention-based fusion demonstrates advances over simple operation-based fusion. However, these fusion works adopt single-scale, i.e., token-level or utterance-level, unimodal representation. Such single-scale fusion is suboptimal because that different modality should be aligned with different granularities. This paper proposes a fusion model named ScaleVLAD to gather multi-Scale representation from text, video, and audio with shared Vectors of Locally Aggregated Descriptors to improve unaligned multimodal sentiment analysis. These shared vectors can be regarded as shared topics to align different modalities. In addition, we propose a self-supervised shifted clustering loss to keep the fused feature differentiation among samples. The backbones are three Transformer encoders corresponding to three modalities, and the aggregated features generated from the fusion module are feed to a Transformer plus a full connection to finish task predictions. Experiments on three popular sentiment analysis benchmarks, IEMOCAP, MOSI, and MOSEI, demonstrate significant gains over baselines.
Code (0)
등록된 구현이 없습니다.
Tasks
Multimodal Sentiment AnalysisSentiment AnalysisMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Multi-scale Cooperative Multimodal Transformers for Multimodal Sentiment Analysis in Videos
Multimodal sentiment analysis in videos is a key task in many real-world applications, which usually requires integrating multimodal streams including visual, verbal and acoustic behaviors. To improve the robustness of m…
Multimodal Sentiment AnalysisSentiment AnalysisPanoSent: A Panoptic Sextuple Extraction Benchmark for Multimodal Conversational Aspect-based Sentiment Analysis
While existing Aspect-based Sentiment Analysis (ABSA) has received extensive effort and advancement, there are still gaps in defining a more holistic research target seamlessly integrating multimodality, conversation con…
Aspect-Based Sentiment AnalysisAspect-Based Sentiment Analysis (ABSA)Language ModellingLarge Language Model+2Interpretable multimodal sentiment analysis based on textual modality descriptions by using large-scale language models
Multimodal sentiment analysis is an important area for understanding the user's internal states. Deep learning methods were effective, but the problem of poor interpretability has gradually gained attention. Previous wor…
Multimodal Sentiment AnalysisSentiment AnalysisSentiMM: A Multimodal Multi-Agent Framework for Sentiment Analysis in Social Media
With the increasing prevalence of multimodal content on social media, sentiment analysis faces significant challenges in effectively processing heterogeneous data and recognizing multi-label emotions. Existing methods of…
Sentiment AnalysisLarge Language Models Meet Text-Centric Multimodal Sentiment Analysis: A Survey
Compared to traditional sentiment analysis, which only considers text, multimodal sentiment analysis needs to consider emotional signals from multimodal sources simultaneously and is therefore more consistent with the wa…
Multimodal Sentiment AnalysisSentiment Analysis