DiffStyleTTS: Diffusion-based Hierarchical Prosody Modeling for Text-to-Speech with Diverse and Controllable Styles

2024-12-04 · Jiaxuan Liu, Zhaoci Liu, Yajun Hu, Yingying Gao, Shilei Zhang, ZhenHua Ling

Human speech exhibits rich and flexible prosodic variations. To address the one-to-many mapping problem from text to prosody in a reasonable and flexible manner, we propose DiffStyleTTS, a multi-speaker acoustic model based on a conditional diffusion module and an improved classifier-free guidance, which hierarchically models speech prosodic features, and controls different prosodic styles to guide prosody prediction. Experiments show that our method outperforms all baselines in naturalness and achieves superior synthesis speed compared to three diffusion-based baselines. Additionally, by adjusting the guiding scale, DiffStyleTTS effectively controls the guidance intensity of the synthetic prosody.

📄 PDF Abstract BibTeX arXiv:2412.03388

Code (0)

등록된 구현이 없습니다.

Tasks

Prosody Prediction text-to-speech Text to Speech

Methods 이 논문이 사용한 방법론

Diffusion Diffusion models generate samples by gradually removing noise from a signal, and their training objective can be expressed as a reweighted variational lower-bound…

SPEED The monocular depth estimation (MDE) is the task of estimating depth from a single frame. This information is an essential knowledge in many computer vision tasks such as scene…

Hierarchical Prosody Modeling for Non-Autoregressive Speech Synthesis

2020-11-12 · Chung-Ming Chien, Hung-Yi Lee

Prosody modeling is an essential component in modern text-to-speech (TTS) frameworks. By explicitly providing prosody features to the TTS model, the style of synthesized utterances can thus be controlled. However, predic…

Speech Synthesis text-to-speech Text to Speech

Fully-hierarchical fine-grained prosody modeling for interpretable speech synthesis

2020-02-06 · Guangzhi Sun, Yu Zhang, Ron J. Weiss, Yuan Cao 외

This paper proposes a hierarchical, fine-grained and interpretable latent variable model for prosody based on the Tacotron 2 text-to-speech model. It achieves multi-resolution modeling of prosody by conditioning finer le…

Disentanglement Speech Synthesis text-to-speech Text to Speech

DiffProsody: Diffusion-based Latent Prosody Generation for Expressive Speech Synthesis with Prosody Conditional Adversarial Training

2023-07-31 · Hyung-Seok Oh, Sang-Hoon Lee, Seong-Whan Lee

Expressive text-to-speech systems have undergone significant advancements owing to prosody modeling, but conventional methods can still be improved. Traditional approaches have relied on the autoregressive method to pred…

Denoising Expressive Speech Synthesis Speech Synthesis text-to-speech+1

Improving Prosody for Cross-Speaker Style Transfer by Semi-Supervised Style Extractor and Hierarchical Modeling in Speech Synthesis

2023-03-14 · Chunyu Qiang, Peng Yang, Hao Che, Ying Zhang 외

Cross-speaker style transfer in speech synthesis aims at transferring a style from source speaker to synthesized speech of a target speaker's timbre. In most previous methods, the synthesized fine-grained prosody feature…

Prosody Prediction Speech Synthesis Style Transfer

CHiVE: Varying Prosody in Speech Synthesis with a Linguistically Driven Dynamic Hierarchical Conditional Variational Network

2019-05-17 · Vincent Wan, Chun-an Chan, Tom Kenter, Jakub Vit 외

The prosodic aspects of speech signals produced by current text-to-speech systems are typically averaged over training material, and as such lack the variety and liveliness found in natural speech. To avoid monotony and …

Decoder Sentence Speech Synthesis text-to-speech+1