Multi-Modal Sequence Fusion via Recursive Attention for Emotion Recognition
Natural human communication is nuanced and inherently multi-modal. Humans possess specialised sensoria for processing vocal, visual, and linguistic, and para-linguistic information, but form an intricately fused percept of the multi-modal data stream to provide a holistic representation. Analysis of emotional content in face-to-face communication is a cognitive task to which humans are particularly attuned, given its sociological importance, and poses a difficult challenge for machine emulation due to the subtlety and expressive variability of cross-modal cues. Inspired by the empirical success of recent so-called End-To-End Memory Networks and related works, we propose an approach based on recursive multi-attention with a shared external memory updated over multiple gated iterations of analysis. We evaluate our model across several large multi-modal datasets and show that global contextualised memory with gated memory update can effectively achieve emotion recognition.
Code (0)
등록된 구현이 없습니다.
Tasks
Emotion RecognitionSimilar Papers 제목 키워드 기반
Recursive Joint Attention for Audio-Visual Fusion in Regression based Emotion Recognition
In video-based emotion recognition (ER), it is important to effectively leverage the complementary relationship among audio (A) and visual (V) modalities, while retaining the intra-modal characteristics of individual mod…
Emotion RecognitionregressionAudio-Visual Person Verification based on Recursive Fusion of Joint Cross-Attention
Person or identity verification has been recently gaining a lot of attention using audio-visual fusion as faces and voices share close associations with each other. Conventional approaches based on audio-visual fusion re…
Recursive Joint Cross-Modal Attention for Multimodal Fusion in Dimensional Emotion Recognition
Though multimodal emotion recognition has achieved significant progress over recent years, the potential of rich synergic relationships across the modalities is not fully exploited. In this paper, we introduce Recursive …
Emotion RecognitionMultimodal Emotion RecognitionAudio-Visual Event Localization via Recursive Fusion by Joint Co-Attention
The major challenge in audio-visual event localization task lies in how to fuse information from multiple modalities effectively. Recent works have shown that attention mechanism is beneficial to the fusion process. In t…
audio-visual event localizationvalidRIV: Recursive Introspection Mask Diffusion Vision Language Model
Mask Diffusion-based Vision Language Models (MDVLMs) have achieved remarkable progress in multimodal understanding tasks. However, these models are unable to correct errors in generated tokens, meaning they lack self-cor…