Unimodal and Multimodal Static Facial Expression Recognition for Virtual Reality Users with EmoHeVRDB
In this study, we explored the potential of utilizing Facial Expression Activations (FEAs) captured via the Meta Quest Pro Virtual Reality (VR) headset for Facial Expression Recognition (FER) in VR settings. Leveraging the EmojiHeroVR Database (EmoHeVRDB), we compared several unimodal approaches and achieved up to 73.02% accuracy for the static FER task with seven emotion categories. Furthermore, we integrated FEA and image data in multimodal approaches, observing significant improvements in recognition accuracy. An intermediate fusion approach achieved the highest accuracy of 80.42%, significantly surpassing the baseline evaluation result of 69.84% reported for EmoHeVRDB's image data. Our study is the first to utilize EmoHeVRDB's unique FEA data for unimodal and multimodal static FER, establishing new benchmarks for FER in VR settings. Our findings highlight the potential of fusing complementary modalities to enhance FER accuracy in VR settings, where conventional image-based methods are severely limited by the occlusion caused by Head-Mounted Displays (HMDs).
Code (1)
Tasks
Facial Expression RecognitionFacial Expression Recognition (FER)Similar Papers 제목 키워드 기반
MMA-DFER: MultiModal Adaptation of unimodal models for Dynamic Facial Expression Recognition in-the-wild
Dynamic Facial Expression Recognition (DFER) has received significant interest in the recent years dictated by its pivotal role in enabling empathic and human-compatible technologies. Achieving robustness towards in-the-…
cross-modal alignmentDynamic Facial Expression RecognitionEmotion RecognitionFacial Expression Recognition+2Multimodal Emotion Recognition and Sentiment Analysis in Multi-Party Conversation Contexts
Emotion recognition and sentiment analysis are pivotal tasks in speech and language processing, particularly in real-world scenarios involving multi-party, conversational data. This paper presents a multimodal approach t…
Emotion RecognitionMultimodal Emotion RecognitionSentiment AnalysisResearch on facial expression recognition based on Multimodal data fusion and neural network
Facial expression recognition is a challenging task when neural network is applied to pattern recognition. Most of the current recognition research is based on single source facial data, which generally has the disadvant…
Facial Expression RecognitionFacial Expression Recognition (FER)Improving Unimodal Inference with Multimodal Transformers
This paper proposes an approach for improving performance of unimodal models with multimodal training. Our approach involves a multi-branch architecture that incorporates unimodal models with a multimodal transformer-bas…
Emotion RecognitionGesture RecognitionHand Gesture RecognitionHand-Gesture Recognition+1A Survey of the Trends in Facial and Expression Recognition Databases and Methods
Automated facial identification and facial expression recognition have been topics of active research over the past few decades. Facial and expression recognition find applications in human-computer interfaces, subject t…
Facial Expression RecognitionFacial Expression Recognition (FER)