M2R2: Missing-Modality Robust emotion Recognition framework with iterative data augmentation
This paper deals with the utterance-level modalities missing problem with uncertain patterns on emotion recognition in conversation (ERC) task. Present models generally predict the speaker's emotions by its current utterance and context, which is degraded by modality missing considerably. Our work proposes a framework Missing-Modality Robust emotion Recognition (M2R2), which trains emotion recognition model with iterative data augmentation by learned common representation. Firstly, a network called Party Attentive Network (PANet) is designed to classify emotions, which tracks all the speakers' states and context. Attention mechanism between speaker with other participants and dialogue topic is used to decentralize dependence on multi-time and multi-party utterances instead of the possible incomplete one. Moreover, the Common Representation Learning (CRL) problem is defined for modality-missing problem. Data imputation methods improved by the adversarial strategy are used here to construct extra features to augment data. Extensive experiments and case studies validate the effectiveness of our methods over baselines for modality-missing emotion recognition on two different datasets.
Code (0)
등록된 구현이 없습니다.
Tasks
Data AugmentationEmotion RecognitionEmotion Recognition in ConversationImputationRepresentation LearningSimilar Papers 제목 키워드 기반
Missing Modality Imagination Network for Emotion Recognition with Uncertain Missing Modalities
Multimodal fusion has been proved to improve emotion recognition performance in previous works. However, in real-world applications, we often encounter the problem of missing modality, and which modalities will be missin…
Emotion RecognitionIncomplete Multimodality-Diffused Emotion Recognition
Human multimodal emotion recognition (MER) aims to perceive and understand human emotions via various heterogeneous modalities, such as language, vision, and acoustic. Compared with unimodality, the complementary informa…
Exploiting modality-invariant feature for robust multimodal emotion recognition with missing modalities
Multimodal emotion recognition leverages complementary information across modalities to gain performance. However, we cannot guarantee that the data of all modalities are always present in practice. In the studies to pre…
Emotion RecognitionMultimodal Emotion RecognitionSemi-supervised Deep Generative Modelling of Incomplete Multi-Modality Emotional Data
There are threefold challenges in emotion recognition. First, it is difficult to recognize human's emotional states only considering a single modality. Second, it is expensive to manually annotate the emotional data. Thi…
Emotion RecognitionImputationADMC: Attention-based Diffusion Model for Missing Modalities Feature Completion
Multimodal emotion and intent recognition is essential for automated human-computer interaction, It aims to analyze users' speech, text, and visual information to predict their emotions or intent. One of the significant …
Intent Recognition