Multimodal Utterance-level Affect Analysis using Visual, Audio and Text Features
The integration of information across multiple modalities and across time is a promising way to enhance the emotion recognition performance of affective systems. Much previous work has focused on instantaneous emotion recognition. The 2018 One-Minute Gradual-Emotion Recognition (OMG-Emotion) challenge, which was held in conjunction with the IEEE World Congress on Computational Intelligence, encouraged participants to address long-term emotion recognition by integrating cues from multiple modalities, including facial expression, audio and language. Intuitively, a multi-modal inference network should be able to leverage information from each modality and their correlations to improve recognition over that achievable by a single modality network. We describe here a multi-modal neural architecture that integrates visual information over time using an LSTM, and combines it with utterance level audio and text cues to recognize human sentiment from multimodal clips. Our model outperforms the unimodal baseline, achieving the concordance correlation coefficients (CCC) of 0.400 on the arousal task, and 0.353 on the valence task.
Code (2)
Tasks
Emotion RecognitionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
A Novel Context-Aware Multimodal Framework for Persian Sentiment Analysis
Most recent works on sentiment analysis have exploited the text modality. However, millions of hours of video recordings posted on social media platforms everyday hold vital unstructured information that can be exploited…
Multimodal Sentiment AnalysisPersian Sentiment AnalysisSentiment AnalysisM3ED: Multi-modal Multi-scene Multi-label Emotional Dialogue Database
The emotional state of a speaker can be influenced by many different factors in dialogues, such as dialogue scene, dialogue topic, and interlocutor stimulus. The currently available data resources to support such multimo…
Cultural Vocal Bursts Intensity PredictionDiversityEmotion RecognitionRethinking Multimodal Sentiment Analysis: A High-Accuracy, Simplified Fusion Architecture
Multimodal sentiment analysis, a pivotal task in affective computing, seeks to understand human emotions by integrating cues from language, audio, and visual signals. While many recent approaches leverage complex attenti…
Emotion ClassificationFeature EngineeringMultimodal Sentiment AnalysisSentiment AnalysisMELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations
Emotion recognition in conversations is a challenging task that has recently gained popularity due to its potential applications. Until now, however, a large-scale multimodal multi-party emotional conversational database…
Dialogue GenerationEmotion RecognitionEmotion Recognition in ConversationMultimodal Sentiment Analysis using Hierarchical Fusion with Context Modeling
Multimodal sentiment analysis is a very actively growing field of research. A promising area of opportunity in this field is to improve the multimodal fusion mechanism. We present a novel feature fusion strategy that pro…
Multimodal Emotion RecognitionMultimodal Sentiment AnalysisSentiment Analysis