Multi Modal Facial Expression Recognition with Transformer-Based Fusion Networks and Dynamic Sampling
Facial expression recognition is an essential task for various applications, including emotion detection, mental health analysis, and human-machine interactions. In this paper, we propose a multi-modal facial expression recognition method that exploits audio information along with facial images to provide a crucial clue to differentiate some ambiguous facial expressions. Specifically, we introduce a Modal Fusion Module (MFM) to fuse audio-visual information, where image and audio features are extracted from Swin Transformer. Additionally, we tackle the imbalance problem in the dataset by employing dynamic data resampling. Our model has been evaluated in the Affective Behavior in-the-wild (ABAW) challenge of CVPR 2023.
Code (0)
등록된 구현이 없습니다.
Tasks
Facial Expression RecognitionFacial Expression Recognition (FER)Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Facial Expression Recognition with Swin Transformer
The task of recognizing human facial expressions plays a vital role in various human-related systems, including health care and medical fields. With the recent success of deep learning and the accessibility of a large am…
Facial Expression RecognitionFacial Expression Recognition (FER)Research on facial expression recognition based on Multimodal data fusion and neural network
Facial expression recognition is a challenging task when neural network is applied to pattern recognition. Most of the current recognition research is based on single source facial data, which generally has the disadvant…
Facial Expression RecognitionFacial Expression Recognition (FER)Smile on the Face, Sadness in the Eyes: Bridging the Emotion Gap with a Multimodal Dataset of Eye and Facial Behaviors
Emotion Recognition (ER) is the process of analyzing and identifying human emotions from sensing data. Currently, the field heavily relies on facial expression recognition (FER) because visual channel conveys rich emotio…
Multimodal Emotion RecognitionFacial Expression RecognitionMulti-Task Multi-Modal Self-Supervised Learning for Facial Expression Recognition
Human communication is multi-modal; e.g., face-to-face interaction involves auditory signals (speech) and visual signals (face movements and hand gestures). Hence, it is essential to exploit multiple modalities when desi…
Emotion ClassificationEmotion Recognition in ConversationFacial Expression RecognitionSelf-Supervised LearningSmile upon the Face but Sadness in the Eyes: Emotion Recognition based on Facial Expressions and Eye Behaviors
Emotion Recognition (ER) is the process of identifying human emotions from given data. Currently, the field heavily relies on facial expression recognition (FER) because facial expressions contain rich emotional cues. Ho…
Emotion RecognitionFacial Expression RecognitionFacial Expression Recognition (FER)Multimodal Emotion Recognition