Seq2Seq2Sentiment: Multimodal Sequence to Sequence Models for Sentiment Analysis
Multimodal machine learning is a core research area spanning the language, visual and acoustic modalities. The central challenge in multimodal learning involves learning representations that can process and relate information from multiple modalities. In this paper, we propose two methods for unsupervised learning of joint multimodal representations using sequence to sequence (Seq2Seq) methods: a \textit{Seq2Seq Modality Translation Model} and a \textit{Hierarchical Seq2Seq Modality Translation Model}. We also explore multiple different variations on the multimodal inputs and outputs of these seq2seq models. Our experiments on multimodal sentiment analysis using the CMU-MOSI dataset indicate that our methods learn informative multimodal representations that outperform the baselines and achieve improved performance on multimodal sentiment analysis, specifically in the Bimodal case where our model is able to improve F1 Score by twelve points. We also discuss future directions for multimodal Seq2Seq methods.
Code (0)
등록된 구현이 없습니다.
Tasks
Multimodal Sentiment AnalysisSentiment AnalysisTranslationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Lightweight Models for Multimodal Sequential Data
Human language encompasses more than just text; it also conveys emotions through tone and gestures. We present a case study of three simple and efficient Transformer-based architectures for predicting sentiment and emoti…
Multimodal Sentiment AnalysisSentiment AnalysisDeep-HOSeq: Deep Higher Order Sequence Fusion for Multimodal Sentiment Analysis
Multimodal sentiment analysis utilizes multiple heterogeneous modalities for sentiment classification. The recent multimodal fusion schemes customize LSTMs to discover intra-modal dynamics and design sophisticated attent…
Multimodal Sentiment AnalysisSentiment AnalysisSentiment ClassificationMulti-scale Cooperative Multimodal Transformers for Multimodal Sentiment Analysis in Videos
Multimodal sentiment analysis in videos is a key task in many real-world applications, which usually requires integrating multimodal streams including visual, verbal and acoustic behaviors. To improve the robustness of m…
Multimodal Sentiment AnalysisSentiment AnalysisCAGMamba: Context-Aware Gated Cross-Modal Mamba Network for Multimodal Sentiment Analysis
Multimodal Sentiment Analysis (MSA) requires effective modeling of cross-modal interactions and contextual dependencies while remaining computationally efficient. Existing fusion approaches predominantly rely on Transfor…
Multimodal Sentiment AnalysisLabel Correction Model for Aspect-based Sentiment Analysis
Aspect-based sentiment analysis includes opinion aspect extraction and aspect sentiment classification. Researchers have attempted to discover the relationship between these two sub-tasks and have proposed the joint mode…
Aspect-Based Sentiment AnalysisAspect-Based Sentiment Analysis (ABSA)Aspect Extractionmodel+3