Continuous Speech Synthesis using per-token Latent Diffusion

2024-10-21 · Arnon Turetzky, Nimrod Shabtay, Slava Shechtman, Hagai Aronowitz, David Haws, Ron Hoory, Avihu Dekel

The success of autoregressive transformer models with discrete tokens has inspired quantization-based approaches for continuous modalities, though these often limit reconstruction quality. We therefore introduce SALAD, a per-token latent diffusion model for zero-shot text-to-speech, that operates on continuous representations. SALAD builds upon the recently proposed expressive diffusion head for image generation, and extends it to generate variable-length outputs. Our approach utilizes semantic tokens for providing contextual information and determining the stopping condition. We suggest three continuous variants for our method, extending popular discrete speech synthesis techniques. Additionally, we implement discrete baselines for each variant and conduct a comparative analysis of discrete versus continuous speech modeling techniques. Our results demonstrate that both continuous and discrete approaches are highly competent, and that SALAD achieves a superior intelligibility score while obtaining speech quality and speaker similarity on par with the ground-truth audio.

📄 PDF Abstract BibTeX arXiv:2410.16048

Code (0)

등록된 구현이 없습니다.

Tasks

Image Generation Quantization Speech Synthesis text-to-speech Text to Speech

Methods 이 논문이 사용한 방법론

Diffusion Diffusion models generate samples by gradually removing noise from a signal, and their training objective can be expressed as a reweighted variational lower-bound…

Latent Diffusion Model Diffusion models applied to latent spaces, which are normally built with (Variational) Autoencoders.

Multimodal Latent Language Modeling with Next-Token Diffusion

2024-12-11 · Yutao Sun, Hangbo Bao, Wenhui Wang, Zhiliang Peng 외

Multimodal generative models require a unified approach to handle both discrete data (e.g., text and code) and continuous data (e.g., image, audio, video). In this work, we propose Latent Language Modeling (LatentLM), wh…

Image Generation Language Modeling Language Modelling multimodal generation+4

Optimality of FSQ Tokens for Continuous Diffusion for Categorical Data with Application to Text-to-Speech

2026-06-08 · Vadim Popov, Wenju Gu, Tasnima Sadekova, Georgii Aparin 외 arxiv

Continuous diffusion for categorical data is a framework belonging to the diffusion family and aiming at generating discrete data. The scientific interest to such models has been constantly increasing these days because …

NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers

2023-04-18 · Kai Shen, Zeqian Ju, Xu Tan, Yanqing Liu 외

Scaling text-to-speech (TTS) to large-scale, multi-speaker, and in-the-wild datasets is important to capture the diversity in human speech such as speaker identities, prosodies, and styles (e.g., singing). Current large …

In-Context Learning Speech Synthesis text-to-speech Text to Speech

Boosting Diffusion Model for Spectrogram Up-sampling in Text-to-speech: An Empirical Study

2024-06-07 · Chong Zhang, Yanqing Liu, Yang Zheng, Sheng Zhao

Scaling text-to-speech (TTS) with autoregressive language model (LM) to large-scale datasets by quantizing waveform into discrete speech tokens is making great progress to capture the diversity and expressiveness in huma…

Diversity Language Modeling Language Modelling text-to-speech+1

Cosh-DiT: Co-Speech Gesture Video Synthesis via Hybrid Audio-Visual Diffusion Transformers

2025-03-13 · Yasheng Sun, Zhiliang Xu, Hang Zhou, Jiazhi Guan 외

Co-speech gesture video synthesis is a challenging task that requires both probabilistic modeling of human gestures and the synthesis of realistic images that align with the rhythmic nuances of speech. To address these c…