paper-with-me

홈 › Papers

Self-attention fusion for audiovisual emotion recognition with incomplete data

2022-01-26 · Kateryna Chumachenko, Alexandros Iosifidis, Moncef Gabbouj

In this paper, we consider the problem of multimodal data analysis with a use case of audiovisual emotion recognition. We propose an architecture capable of learning from raw data and describe three variants of it with distinct modality fusion mechanisms. While most of the previous works consider the ideal scenario of presence of both modalities at all times during inference, we evaluate the robustness of the model in the unconstrained settings where one modality is absent or noisy, and propose a method to mitigate these limitations in a form of modality dropout. Most importantly, we find that following this approach not only improves performance drastically under the absence/noisy representations of one modality, but also improves the performance in a standard ideal setting, outperforming the competing methods.

📄 PDF Abstract BibTeX arXiv:2201.11095

Code (2)

katerynaCh/multimodal-emotion-recognition 공식 구현 pytorch
shravan-18/AVTCA pytorch

Tasks

Emotion RecognitionFacial Emotion Recognition

Similar Papers 제목 키워드 기반

Continuous-Time Audiovisual Fusion with Recurrence vs. Attention for In-The-Wild Affect Recognition

2022-03-24 · Vincent Karas, Mani Kumar Tellamekala, Adria Mallol-Ragolta, Michel Valstar 외

In this paper, we present our submission to 3rd Affective Behavior Analysis in-the-wild (ABAW) challenge. Learningcomplex interactions among multimodal sequences is critical to recognise dimensional affect from in-the-wi…

Arousal EstimationEmotion RecognitionMultimodal Emotion Recognition

A vector quantized masked autoencoder for audiovisual speech emotion recognition

2023-05-05 · Samir Sadok, Simon Leglaive, Renaud Séguier

An important challenge in emotion recognition is to develop methods that can leverage unlabeled training data. In this paper, we propose the VQ-MAE-AV model, a self-supervised multimodal model that leverages masked autoe…

Contrastive LearningEmotion RecognitionRepresentation LearningSelf-Supervised Learning+1

COLD Fusion: Calibrated and Ordinal Latent Distribution Fusion for Uncertainty-Aware Multimodal Emotion Recognition

2022-06-12 · Mani Kumar Tellamekala, Shahin Amiriparian, Björn W. Schuller, Elisabeth André 외

Automatically recognising apparent emotions from face and voice is hard, in part because of various sources of uncertainty, including in the input data and the labels used in a machine learning framework. This paper intr…

Emotion RecognitionMultimodal Emotion Recognition

Multitask Learning and Multistage Fusion for Dimensional Audiovisual Emotion Recognition

2020-02-26 · ICASSP 2020 4 · Bagus Tris Atmaja, Masato Akagi

Due to its ability to accurately predict emotional state using multimodal features, audiovisual emotion recognition has recently gained more interest from researchers. This paper proposes two methods to predict emotional…

AttributeEmotion Recognition

Relational graph-driven differential denoising and diffusion attention fusion for multimodal conversation emotion recognition

2026-03-22 · Ying Liu, Yuntao Shou, Wei Ai, Tao Meng 외 arxiv

In real-world scenarios, audio and video signals are often subject to environmental noise and limited acquisition conditions, resulting in extracted features containing excessive noise. Furthermore, there is an imbalance…

Emotion Recognition