Leveraging Acoustic Cues and Paralinguistic Embeddings to Detect Expression from Voice
Millions of people reach out to digital assistants such as Siri every day, asking for information, making phone calls, seeking assistance, and much more. The expectation is that such assistants should understand the intent of the users query. Detecting the intent of a query from a short, isolated utterance is a difficult task. Intent cannot always be obtained from speech-recognized transcriptions. A transcription driven approach can interpret what has been said but fails to acknowledge how it has been said, and as a consequence, may ignore the expression present in the voice. Our work investigates whether a system can reliably detect vocal expression in queries using acoustic and paralinguistic embedding. Results show that the proposed method offers a relative equal error rate (EER) decrease of 60% compared to a bag-of-word based system, corroborating that expression is significantly represented by vocal attributes, rather than being purely lexical. Addition of emotion embedding helped to reduce the EER by 30% relative to the acoustic embedding, demonstrating the relevance of emotion in expressive voice.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Paralinguistics-Enhanced Large Language Modeling of Spoken Dialogue
Large Language Models (LLMs) have demonstrated superior abilities in tasks such as chatting, reasoning, and question-answering. However, standard LLMs may ignore crucial paralinguistic information, such as sentiment, emo…
AttributeLanguage ModelingLanguage ModellingQuestion Answering+3Teach me with a Whisper: Enhancing Large Language Models for Analyzing Spoken Transcripts using Speech Embeddings
Speech data has rich acoustic and paralinguistic information with important cues for understanding a speaker's tone, emotion, and intent, yet traditional large language models such as BERT do not incorporate this informa…
Knowledge DistillationLanguage ModelingLanguage ModellingLearning Paralinguistic Features from Audiobooks through Style Voice Conversion
Paralinguistics, the non-lexical components of speech, play a crucial role in human-human interaction. Models designed to recognize paralinguistic information, particularly speech emotion and style, are difficult to trai…
Emotion RecognitionStyle DetectionVoice ConversionDo Audio LLMs Listen or Read? Analyzing and Mitigating Paralinguistic Failures with VoxParadox
Audio large language models (Audio LLMs) demonstrate strong performance on speech understanding tasks, yet their ability to understand paralinguistic information remains limited. To systematically quantify this issue, we…
Speech SynthesisSpeechSense: A Paralinguistic-Focused Dataset for Fine-Grained Speech Sentiment Analysis
Recent advances in AI have revolutionized speech processing, yet effective speech understanding requires discerning not just what is said, but how it is said. Speech Sentiment Analysis plays a critical role in decoding t…
Sentiment AnalysisSpeech RecognitionSpeech Synthesis