Multimodal Continuous Turn-Taking Prediction Using Multiscale RNNs
In human conversational interactions, turn-taking exchanges can be coordinated using cues from multiple modalities. To design spoken dialog systems that can conduct fluid interactions it is desirable to incorporate cues from separate modalities into turn-taking models. We propose that there is an appropriate temporal granularity at which modalities should be modeled. We design a multiscale RNN architecture to model modalities at separate timescales in a continuous manner. Our results show that modeling linguistic and acoustic features at separate temporal rates can be beneficial for turn-taking modeling. We also show that our approach can be used to incorporate gaze features into turn-taking models.
Code (1)
Tasks
PredictionSimilar Papers 제목 키워드 기반
Real-TurnTurk: A Multimodal Turkish Corpus for Turn-Taking Prediction
Turn-taking is a basic organizational feature of human conversation and remains difficult to model in natural, synchronous dialog systems. While existing research has explored multimodal approaches and large language mod…
Binary ClassificationGated Multimodal Fusion with Contrastive Learning for Turn-taking Prediction in Human-robot Dialogue
Turn-taking, aiming to decide when the next speaker can start talking, is an essential component in building human-robot spoken dialogue systems. Previous studies indicate that multimodal cues can facilitate this challen…
Contrastive LearningData AugmentationSentenceSpoken Dialogue SystemsVisual Cues Enhance Predictive Turn-Taking for Two-Party Human Interaction
Turn-taking is richly multimodal. Predictive turn-taking models (PTTMs) facilitate naturalistic human-robot interaction, yet most rely solely on speech. We introduce MM-VAP, a multimodal PTTM which combines speech with v…
Multimodal Transformer Models for Turn-taking Prediction: Effects on Conversational Dynamics of Human-Agent Interaction during Cooperative Gameplay
This study investigates multimodal turn-taking prediction within human-agent interactions (HAI), particularly focusing on cooperative gaming environments. It comprises both model development and subsequent user study, ai…
Spoken Dialogue SystemsInvestigating Speech Features for Continuous Turn-Taking Prediction Using LSTMs
For spoken dialog systems to conduct fluid conversational interactions with users, the systems must be sensitive to turn-taking cues produced by a user. Models should be designed so that effective decisions can be made a…