MVP: Multimodal Emotion Recognition based on Video and Physiological Signals
Human emotions entail a complex set of behavioral, physiological and cognitive changes. Current state-of-the-art models fuse the behavioral and physiological components using classic machine learning, rather than recent deep learning techniques. We propose to fill this gap, designing the Multimodal for Video and Physio (MVP) architecture, streamlined to fuse video and physiological signals. Differently then others approaches, MVP exploits the benefits of attention to enable the use of long input sequences (1-2 minutes). We have studied video and physiological backbones for inputting long sequences and evaluated our method with respect to the state-of-the-art. Our results show that MVP outperforms former methods for emotion recognition based on facial videos, EDA, and ECG/PPG.
Code (0)
등록된 구현이 없습니다.
Tasks
Emotion RecognitionMultimodal Emotion RecognitionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Interpretable Multimodal Emotion Recognition using Facial Features and Physiological Signals
This paper aims to demonstrate the importance and feasibility of fusing multimodal information for emotion recognition. It introduces a multimodal framework for emotion understanding by fusing the information from visual…
Emotion ClassificationEmotion RecognitionFeature ImportanceMultimodal Emotion RecognitionEmotion recognition based on multi-modal electrophysiology multi-head attention Contrastive Learning
Emotion recognition is an important research direction in artificial intelligence, helping machines understand and adapt to human emotional states. Multimodal electrophysiological(ME) signals, such as EEG, GSR, respirati…
Contrastive LearningEEGEmotion RecognitionMultimodal Emotion RecognitionPHemoNet: A Multimodal Network for Physiological Signals
Emotion recognition is essential across numerous fields, including medical applications and brain-computer interface (BCI). Emotional responses include behavioral reactions, such as tone of voice and body movement, and c…
Brain Computer InterfaceEEGElectroencephalogram (EEG)Emotion Recognition+2A Survey on Physiological Signal Based Emotion Recognition
Physiological Signals are the most reliable form of signals for emotion recognition, as they cannot be controlled deliberately by the subject. Existing review papers on emotion recognition based on physiological signals …
Emotion RecognitionSurveyMultimodal Emotion Recognition by Fusing Video Semantic in MOOC Learning Scenarios
In the Massive Open Online Courses (MOOC) learning scenario, the semantic information of instructional videos has a crucial impact on learners' emotional state. Learners mainly acquire knowledge by watching instructional…
Emotion RecognitionLanguage ModellingLarge Language ModelMultimodal Emotion Recognition+1