Boosting Diffusion Model for Spectrogram Up-sampling in Text-to-speech: An Empirical Study

2024-06-07 · Chong Zhang, Yanqing Liu, Yang Zheng, Sheng Zhao

Scaling text-to-speech (TTS) with autoregressive language model (LM) to large-scale datasets by quantizing waveform into discrete speech tokens is making great progress to capture the diversity and expressiveness in human speech, but the speech reconstruction quality from discrete speech token is far from satisfaction depending on the compressed speech token compression ratio. Generative diffusion models trained with score-matching loss and continuous normalized flow trained with flow-matching loss have become prominent in generation of images as well as speech. LM based TTS systems usually quantize speech into discrete tokens and generate these tokens autoregressively, and finally use a diffusion model to up sample coarse-grained speech tokens into fine-grained codec features or mel-spectrograms before reconstructing into waveforms with vocoder, which has a high latency and is not realistic for real time speech applications. In this paper, we systematically investigate varied diffusion models for up sampling stage, which is the main bottleneck for streaming synthesis of LM and diffusion-based architecture, we present the model architecture, objective and subjective metrics to show quality and efficiency improvement.

📄 PDF Abstract BibTeX arXiv:2406.04633

Code (0)

등록된 구현이 없습니다.

Tasks

DiversityLanguage ModelingLanguage Modelling text-to-speech Text to Speech

Methods 이 논문이 사용한 방법론

Diffusion Diffusion models generate samples by gradually removing noise from a signal, and their training objective can be expressed as a reweighted variational lower-bound…

FastDiff: A Fast Conditional Diffusion Model for High-Quality Speech Synthesis

2022-04-21 · Rongjie Huang, Max W. Y. Lam, Jun Wang, Dan Su 외

Denoising diffusion probabilistic models (DDPMs) have recently achieved leading performances in many generative tasks. However, the inherited iterative sampling process costs hindered their applications to speech synthes…

Denoising GPU Speech Synthesis text-to-speech+3

Diff-TTS: A Denoising Diffusion Model for Text-to-Speech

2021-04-03 · Myeonghun Jeong, Hyeongju Kim, Sung Jun Cheon, Byoung Jin Choi 외

Although neural text-to-speech (TTS) models have attracted a lot of attention and succeeded in generating human-like speech, there is still room for improvements to its naturalness and architectural efficiency. In this w…

Denoising GPU Speech Synthesis text-to-speech+1

Boosting Fast and High-Quality Speech Synthesis with Linear Diffusion

2023-06-09 · Haogeng Liu, Tao Wang, Jie Cao, Ran He 외

Denoising Diffusion Probabilistic Models have shown extraordinary ability on various generative tasks. However, their slow inference speed renders them impractical in speech synthesis. This paper proposes a linear diffus…

Denoising Speech Synthesis

ProDiff: Progressive Fast Diffusion Model For High-Quality Text-to-Speech

2022-07-13 · Rongjie Huang, Zhou Zhao, Huadai Liu, Jinglin Liu 외

Denoising GPU Knowledge Distillation Speech Synthesis+4

VoiceFlow: Efficient Text-to-Speech with Rectified Flow Matching

2023-09-10 · Yiwei Guo, Chenpeng Du, Ziyang Ma, Xie Chen 외

Although diffusion models in text-to-speech have become a popular choice due to their strong generative ability, the intrinsic complexity of sampling from diffusion models harms their efficiency. Alternatively, we propos…

text-to-speech Text to Speech