Chain-of-Thought Training for Open E2E Spoken Dialogue Systems
Unlike traditional cascaded pipelines, end-to-end (E2E) spoken dialogue systems preserve full differentiability and capture non-phonemic information, making them well-suited for modeling spoken interactions. However, existing E2E approaches often require large-scale training data and generates responses lacking semantic coherence. We propose a simple yet effective strategy leveraging a chain-of-thought (CoT) formulation, ensuring that training on conversational data remains closely aligned with the multimodal language model (LM)'s pre-training on speech recognition~(ASR), text-to-speech synthesis (TTS), and text LM tasks. Our method achieves over 1.5 ROUGE-1 improvement over the baseline, successfully training spoken dialogue systems on publicly available human-human conversation datasets, while being compute-efficient enough to train on just 300 hours of public human-human conversation data, such as the Switchboard. We will publicly release our models and training code.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage Modellingspeech-recognitionSpeech RecognitionSpeech SynthesisSpoken Dialogue Systemstext-to-speechText to SpeechText-To-Speech SynthesisSimilar Papers 제목 키워드 기반
Leveraging Chain of Thought towards Empathetic Spoken Dialogue without Corresponding Question-Answering Data
Empathetic dialogue is crucial for natural human-computer interaction, allowing the dialogue system to respond in a more personalized and emotionally aware manner, improving user satisfaction and engagement. The emergenc…
Dialogue GenerationQuestion AnsweringChain-of-Thought Reasoning in Streaming Full-Duplex End-to-End Spoken Dialogue Systems
Most end-to-end (E2E) spoken dialogue systems (SDS) rely on voice activity detection (VAD) for turn-taking, but VAD fails to distinguish between pauses and turn completions. Duplex SDS models address this by predicting o…
Activity DetectionReasoning in Conversation: Solving Subjective Tasks through Dialogue Simulation for Large Language Models
Large Language Models (LLMs) have achieved remarkable performance in objective tasks such as open-domain question answering and mathematical reasoning, which can often be solved through recalling learned factual knowledg…
Dark Humor DetectionDialogue GenerationHumor DetectionMathematical Reasoning+2Enabling Conversational Behavior Reasoning Capabilities in Full-Duplex Speech
Human conversation is organized by an implicit chain of thoughts that manifests as timed speech acts. Capturing this causal pathway is key to building natural full-duplex interactive systems. We introduce a framework tha…
Causal InferenceInjecting Salesperson's Dialogue Strategies in Large Language Models with Chain-of-Thought Reasoning
Recent research in dialogue systems and corpora has focused on two main categories: task-oriented (TOD) and open-domain (chit-chat) dialogues. TOD systems help users accomplish specific tasks, while open-domain systems a…