When can I Speak? Predicting initiation points for spoken dialogue agents
Current spoken dialogue systems initiate their turns after a long period of silence (700-1000ms), which leads to little real-time feedback, sluggish responses, and an overall stilted conversational flow. Humans typically respond within 200ms and successfully predicting initiation points in advance would allow spoken dialogue agents to do the same. In this work, we predict the lead-time to initiation using prosodic features from a pre-trained speech representation model (wav2vec 1.0) operating on user audio and word features from a pre-trained language model (GPT-2) operating on incremental transcriptions. To evaluate errors, we propose two metrics w.r.t. predicted and true lead times. We train and evaluate the models on the Switchboard Corpus and find that our method outperforms features from prior work on both metrics and vastly outperforms the common approach of waiting for 700ms of silence.
Code (1)
Tasks
Language ModelingLanguage ModellingSpoken Dialogue SystemsSimilar Papers 제목 키워드 기반
Improving End-of-turn Detection in Spoken Dialogues by Detecting Speaker Intentions as a Secondary Task
This work focuses on the use of acoustic cues for modeling turn-taking in dyadic spoken dialogues. Previous work has shown that speaker intentions (e.g., asking a question, uttering a backchannel, etc.) can influence tur…
Initiations and Interruptions in a Spoken Dialog System
Joint Modeling of Content and Discourse Relations in Dialogues
We present a joint modeling approach to identify salient discussion points in spoken meetings as well as to label the discourse relations between speaker turns. A variation of our model is also discussed when discourse r…
Relation PredictionLarge Language Models Know What To Say But Not When To Speak
Turn-taking is a fundamental mechanism in human communication that ensures smooth and coherent verbal interactions. Recent advances in Large Language Models (LLMs) have motivated their use in improving the turn-taking ca…
Spoken Dialogue SystemsThe importance of fillers for text representations of speech transcripts
While being an essential component of spoken language, fillers (e.g."um" or "uh") often remain overlooked in Spoken Language Understanding (SLU) tasks. We explore the possibility of representing them with deep contextual…
Spoken Language Understanding