Enhancing Speech Instruction Understanding and Disambiguation in Robotics via Speech Prosody
Enabling robots to accurately interpret and execute spoken language instructions is essential for effective human-robot collaboration. Traditional methods rely on speech recognition to transcribe speech into text, often discarding crucial prosodic cues needed for disambiguating intent. We propose a novel approach that directly leverages speech prosody to infer and resolve instruction intent. Predicted intents are integrated into large language models via in-context learning to disambiguate and select appropriate task plans. Additionally, we present the first ambiguous speech dataset for robotics, designed to advance research in speech disambiguation. Our method achieves 95.79% accuracy in detecting referent intents within an utterance and determines the intended task plan of ambiguous instructions with 71.96% accuracy, demonstrating its potential to significantly improve human-robot communication.
Code (0)
등록된 구현이 없습니다.
Tasks
In-Context Learningspeech-recognitionSpeech RecognitionSimilar Papers 제목 키워드 기반
From Speech-to-Spatial: Grounding Utterances on A Live Shared View with Augmented Reality
We introduce Speech-to-Spatial, a referent disambiguation framework that converts verbal remote-assistance instructions into spatially grounded AR guidance. Unlike prior systems that rely on additional cues (e.g., gestur…
BatonVoice: An Operationalist Framework for Enhancing Controllable Speech Synthesis with Linguistic Intelligence from LLMs
The rise of Large Language Models (LLMs) is reshaping multimodel models, with speech synthesis being a prominent application. However, existing approaches often underutilize the linguistic intelligence of these models, t…
Speech SynthesisIntegrating Disambiguation and User Preferences into Large Language Models for Robot Motion Planning
This paper presents a framework that can interpret humans' navigation commands containing temporal elements and directly translate their natural language instructions into robot motion planning. Central to our framework …
Motion PlanningDEBATE: A Dataset for Disentangling Textual Ambiguity in Mandarin Through Speech
Despite extensive research on textual and visual disambiguation, disambiguation through speech (DTS) remains underexplored. This is largely due to the lack of high-quality datasets that pair spoken sentences with richly …
UniverSLU: Universal Spoken Language Understanding for Diverse Tasks with Natural Language Instructions
Recent studies leverage large language models with multi-tasking capabilities, using natural language prompts to guide the model's behavior and surpassing performance of task-specific models. Motivated by this, we ask: c…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Multi-Task Learningspeech-recognition+2