Stimulus Modality Matters: Impact of Perceptual Evaluations from Different Modalities on Speech Emotion Recognition System Performance
Speech Emotion Recognition (SER) systems rely on speech input and emotional labels annotated by humans. However, various emotion databases collect perceptional evaluations in different ways. For instance, the IEMOCAP dataset uses video clips with sounds for annotators to provide their emotional perceptions. However, the most significant English emotion dataset, the MSP-PODCAST, only provides speech for raters to choose the emotional ratings. Nevertheless, using speech as input is the standard approach to training SER systems. Therefore, the open question is the emotional labels elicited by which scenarios are the most effective for training SER systems. We comprehensively compare the effectiveness of SER systems trained with labels elicited by different modality stimuli and evaluate the SER systems on various testing conditions. Also, we introduce an all-inclusive label that combines all labels elicited by various modalities. We show that using labels elicited by voice-only stimuli for training yields better performance on the test set, whereas labels elicited by voice-only stimuli.
Code (0)
등록된 구현이 없습니다.
Tasks
Emotion RecognitionSpeech Emotion RecognitionSimilar Papers 제목 키워드 기반
Perceptual implications of simplifying geometrical acoustics models for Ambisonics-based binaural reverberation
Different methods can be employed to render virtual reverberation, often requiring substantial information about the room's geometry and the acoustic characteristics of the surfaces. However, fully comprehensive approach…
The Brain Uses Reliability of Stimulus Information when Making Perceptual Decisions
In simple perceptual decisions the brain has to identify a stimulus based on noisy sensory samples from the stimulus. Basic statistical considerations state that the reliability of the stimulus information, i.e., the amo…
Decision MakingMEG Evidence That Modality-Independent Conceptual Representations Encode Visual but Not Lexical Representations
The semantic knowledge stored in our brains can be accessed from different stimulus modalities. For example, a picture of a cat and the word "cat" both engage similar conceptual representations. While existing research h…
Hebbian Plasticity for Improving Perceptual Decisions
Shibata et al. reported that humans could learn to repeatedly evoke a stimulus-associated functional magnetic resonance imaging (fMRI) activity pattern in visual areas V1/V2 through which visual perceptual learning was a…
Multi-scale variability in neuronal competition
We examine whether a single biophysical cortical circuit model can explain both spiking and perceptual variability. We consider perceptual rivalry, which provides a window into intrinsic neural processing since neural ac…
Open-Ended Question Answering