Leveraging TCN and Transformer for effective visual-audio fusion in continuous emotion recognition
Human emotion recognition plays an important role in human-computer interaction. In this paper, we present our approach to the Valence-Arousal (VA) Estimation Challenge, Expression (Expr) Classification Challenge, and Action Unit (AU) Detection Challenge of the 5th Workshop and Competition on Affective Behavior Analysis in-the-wild (ABAW). Specifically, we propose a novel multi-modal fusion model that leverages Temporal Convolutional Networks (TCN) and Transformer to enhance the performance of continuous emotion recognition. Our model aims to effectively integrate visual and audio information for improved accuracy in recognizing emotions. Our model outperforms the baseline and ranks 3 in the Expression Classification challenge.
Code (1)
Tasks
Emotion RecognitionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Dynamic Multi-Target Fusion for Efficient Audio-Visual Navigation
Audiovisual embodied navigation enables robots to locate audio sources by dynamically integrating visual observations from onboard sensors with the auditory signals emitted by the target. The core challenge lies in effec…
Visual NavigationUnveiling the Power of Audio-Visual Early Fusion Transformers with Dense Interactions through Masked Modeling
Humans possess a remarkable ability to integrate auditory and visual information, enabling a deeper understanding of the surrounding environment. This early fusion of audio and visual cues, demonstrated through cognitive…
Audio-Visual Speech Recognition based on Regulated Transformer and Spatio-Temporal Fusion Strategy for Driver Assistive Systems
This article presents a research methodology for audio-visual speech recognition (AVSR) in driver assistive systems. These systems necessitate ongoing interaction with drivers while driving through voice control for safe…
Audio-Visual Speech RecognitionLipreadingLip Readingspeech-recognition+2AVT: Audio-Video Transformer for Multimodal Action Recognition
Action recognition is an essential field for video understanding. To learn from heterogeneous data sources effectively, in this work, we propose a novel multimodal action recognition approach termed Audio-Video Transform…
Action RecognitionAudio ClassificationContrastive LearningMulti-modal Classification+1Attentive Fusion Enhanced Audio-Visual Encoding for Transformer Based Robust Speech Recognition
Audio-visual information fusion enables a performance improvement in speech recognition performed in complex acoustic scenarios, e.g., noisy environments. It is required to explore an effective audio-visual fusion strate…
Robust Speech Recognitionspeech-recognitionSpeech Recognition