Prompting Audios Using Acoustic Properties For Emotion Representation
Emotions lie on a continuum, but current models treat emotions as a finite valued discrete variable. This representation does not capture the diversity in the expression of emotion. To better represent emotions we propose the use of natural language descriptions (or prompts). In this work, we address the challenge of automatically generating these prompts and training a model to better learn emotion representations from audio and prompt pairs. We use acoustic properties that are correlated to emotion like pitch, intensity, speech rate, and articulation rate to automatically generate prompts i.e. 'acoustic prompts'. We use a contrastive learning objective to map speech to their respective acoustic prompts. We evaluate our model on Emotion Audio Retrieval and Speech Emotion Recognition. Our results show that the acoustic prompts significantly improve the model's performance in EAR, in various Precision@K metrics. In SER, we observe a 3.8% relative accuracy improvement on the Ravdess dataset.
Code (0)
등록된 구현이 없습니다.
Tasks
Contrastive LearningDiversityEmotion RecognitionRetrievalSpeech Emotion RecognitionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
OmniVox: Zero-Shot Emotion Recognition with Omni-LLMs
The use of omni-LLMs (large language models that accept any modality as input), particularly for multimodal cognitive state tasks involving speech, is understudied. We present OmniVox, the first systematic evaluation of …
Emotion RecognitionPlug-and-Play Emotion Graphs for Compositional Prompting in Zero-Shot Speech Emotion Recognition
Large audio-language models (LALMs) exhibit strong zero-shot performance across speech tasks but struggle with speech emotion recognition (SER) due to weak paralinguistic modeling and limited cross-modal reasoning. We pr…
Speech Emotion RecognitionDeep Net Features for Complex Emotion Recognition
This paper investigates the influence of different acoustic features, audio-events based features and automatic speech translation based lexical features in complex emotion recognition such as curiosity. Pretrained netwo…
automatic-speech-translationEmotion RecognitionTranslationSpatio-temporal Latent Representations for the Analysis of Acoustic Scenes in-the-wild
In the field of acoustic scene analysis, this paper presents a novel approach to find spatio-temporal latent representations from in-the-wild audio data. By using WE-LIVE, an in-house collected dataset that includes audi…
Information RetrievalSelf-Supervised LearningPerceptual Loss based Speech Denoising with an ensemble of Audio Pattern Recognition and Self-Supervised Models
Deep learning based speech denoising still suffers from the challenge of improving perceptual quality of enhanced signals. We introduce a generalized framework called Perceptual Ensemble Regularization Loss (PERL) built …
DenoisingEmotion ClassificationMulti-Task LearningSpeech Denoising