SpeakStream: Streaming Text-to-Speech with Interleaved Data

2025-05-25 · Richard He Bai, Zijin Gu, Tatiana Likhomanenko, Navdeep Jaitly

The latency bottleneck of traditional text-to-speech (TTS) systems fundamentally hinders the potential of streaming large language models (LLMs) in conversational AI. These TTS systems, typically trained and inferenced on complete utterances, introduce unacceptable delays, even with optimized inference speeds, when coupled with streaming LLM outputs. This is particularly problematic for creating responsive conversational agents where low first-token latency is critical. In this paper, we present SpeakStream, a streaming TTS system that generates audio incrementally from streaming text using a decoder-only architecture. SpeakStream is trained using a next-step prediction loss on interleaved text-speech data. During inference, it generates speech incrementally while absorbing streaming input text, making it particularly suitable for cascaded conversational AI agents where an LLM streams text to a TTS system. Our experiments demonstrate that SpeakStream achieves state-of-the-art latency results in terms of first-token latency while maintaining the quality of non-streaming TTS systems.

📄 PDF Abstract BibTeX arXiv:2505.19206

Code (0)

등록된 구현이 없습니다.

Tasks

Decoder text-to-speech Text to Speech

Interleaved Speech-Text Language Models are Simple Streaming Text to Speech Synthesizers

2024-12-20 · Yifan Yang, Ziyang Ma, Shujie Liu, Jinyu Li 외

This paper introduces Interleaved Speech-Text Language Model (IST-LM) for streaming zero-shot Text-to-Speech (TTS). Unlike many previous approaches, IST-LM is directly trained on interleaved sequences of text and speech …

Language Modeling Language Modelling text-to-speech Text to Speech

Conv-Transformer Transducer: Low Latency, Low Frame Rate, Streamable End-to-End Speech Recognition

2020-08-13 · Wenyong Huang, Wenchao Hu, Yu Ting Yeung, Xiao Chen

Transformer has achieved competitive performance against state-of-the-art end-to-end models in automatic speech recognition (ASR), and requires significantly less training time than RNN-based models. The original Transfo…

Automatic Speech Recognition Automatic Speech Recognition (ASR)Decoder speech-recognition+1

StarTSE: Towards Streaming Target Speaker Extraction via Chunk-wise Interleaved Splicing of Autoregressive Language Model

2026-04-21 · Shuhai Peng, Hui Lu, Jinjiang Liu, Liyang Chen 외 arxiv

While generative models have set new benchmarks for Target Speaker Extraction (TSE), their inherent reliance on global context precludes deployment in real-time applications. Direct adaptation to streaming scenarios ofte…

Streaming T5-based Text-to-Speech Synthesis with Limited Lookahead

2026-06-20 · Muyang Du, Jason Roche, Junjie Lai arxiv

Streaming text-to-speech synthesis in cascaded LLM-TTS systems still faces latency challenges as most TTS models require full context before initiating generation. We present S5-TTS, a streaming variant of T5-TTS that en…

Text-To-Speech Synthesis

StreamMel: Real-Time Zero-shot Text-to-Speech via Interleaved Continuous Autoregressive Modeling

2025-06-14 · Hui Wang, Yifan Yang, Shujie Liu, Jinyu Li 외

Recent advances in zero-shot text-to-speech (TTS) synthesis have achieved high-quality speech generation for unseen speakers, but most systems remain unsuitable for real-time applications because of their offline design.…

text-to-speech Text to Speech