FAF: A novel multimodal emotion recognition approach integrating face, body and text
Multimodal emotion analysis performed better in emotion recognition depending on more comprehensive emotional clues and multimodal emotion dataset. In this paper, we developed a large multimodal emotion dataset, named "HED" dataset, to facilitate the emotion recognition task, and accordingly propose a multimodal emotion recognition method. To promote recognition accuracy, "Feature After Feature" framework was used to explore crucial emotional information from the aligned face, body and text samples. We employ various benchmarks to evaluate the "HED" dataset and compare the performance with our method. The results show that the five classification accuracy of the proposed multimodal fusion method is about 83.75%, and the performance is improved by 1.83%, 9.38%, and 21.62% respectively compared with that of individual modalities. The complementarity between each channel is effectively used to improve the performance of emotion recognition. We had also established a multimodal online emotion prediction platform, aiming to provide free emotion prediction to more users.
Code (0)
등록된 구현이 없습니다.
Tasks
Emotion RecognitionMultimodal Emotion RecognitionSimilar Papers 제목 키워드 기반
Emotion Detection through Body Gesture and Face
The project leverages advanced machine and deep learning techniques to address the challenge of emotion recognition by focusing on non-facial cues, specifically hands, body gestures, and gestures. Traditional emotion rec…
Emotion ClassificationEmotion RecognitionPose EstimationA Multimodal Emotion Recognition System: Integrating Facial Expressions, Body Movement, Speech, and Spoken Language
Traditional psychological evaluations rely heavily on human observation and interpretation, which are prone to subjectivity, bias, fatigue, and inconsistency. To address these limitations, this work presents a multimodal…
DiagnosticEmotion RecognitionMultimodal Emotion RecognitionMultimodal Affect Recognition using Kinect
Affect (emotion) recognition has gained significant attention from researchers in the past decade. Emotion-aware computer systems and devices have many applications ranging from interactive robots, intelligent online tut…
Emotion RecognitionMultimodal Group Emotion Recognition In-the-Wild Towards a Privacy-Safe Non-Individual Approach
This thesis addresses group emotion recognition (GER) in-the-wild with a focus on privacy preservation. Unlike traditional emotion recognition methods that rely on individual-level cues such as face, gaze, or voice analy…
Emotion ClassificationEmotion RecognitionData AugmentationSeeing is Believing: Emotion-Aware Audio-Visual Language Modeling for Expressive Speech Generation
We present an Audio-Visual Language Model (AVLM) for expressive speech generation by integrating full-face visual cues into a pre-trained expressive speech model. We explore multiple visual encoders and multimodal fusion…
Emotion Recognition