Autoregressive Speech Synthesis without Vector Quantization
We present MELLE, a novel continuous-valued token based language modeling approach for text-to-speech synthesis (TTS). MELLE autoregressively generates continuous mel-spectrogram frames directly from text condition, bypassing the need for vector quantization, which is typically designed for audio compression and sacrifices fidelity compared to continuous representations. Specifically, (i) instead of cross-entropy loss, we apply regression loss with a proposed spectrogram flux loss function to model the probability distribution of the continuous-valued tokens; (ii) we have incorporated variational inference into MELLE to facilitate sampling mechanisms, thereby enhancing the output diversity and model robustness. Experiments demonstrate that, compared to the two-stage codec language model VALL-E and its variants, the single-stage MELLE mitigates robustness issues by avoiding the inherent flaws of sampling vector-quantized codes, achieves superior performance across multiple metrics, and, most importantly, offers a more streamlined paradigm. The demos of our work are provided at https://aka.ms/melle.
Code (0)
등록된 구현이 없습니다.
Tasks
Audio CompressionDiversityLanguage ModelingLanguage ModellingQuantizationSpeech Synthesistext-to-speechText to SpeechText-To-Speech SynthesisVariational InferenceMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
MAG: Multi-Modal Aligned Autoregressive Co-Speech Gesture Generation without Vector Quantization
This work focuses on full-body co-speech gesture generation. Existing methods typically employ an autoregressive model accompanied by vector-quantized tokens for gesture generation, which results in information loss and …
Gesture GenerationQuantizationContinuous Autoregressive Modeling with Stochastic Monotonic Alignment for Speech Synthesis
We propose a novel autoregressive modeling approach for speech synthesis, combining a variational autoencoder (VAE) with a multi-modal latent space and an autoregressive model that uses Gaussian Mixture Models (GMM) as t…
QuantizationSpeech SynthesisSpectral Codecs: Improving Non-Autoregressive Speech Synthesis with Spectrogram-Based Audio Codecs
Historically, most speech models in machine-learning have used the mel-spectrogram as a speech representation. Recently, discrete audio tokens produced by neural audio codecs have become a popular alternate speech repres…
QuantizationSpeech Synthesistext-to-speechText to SpeechHALL-E: Hierarchical Neural Codec Language Model for Minute-Long Zero-Shot Text-to-Speech Synthesis
Recently, Text-to-speech (TTS) models based on large language models (LLMs) that translate natural language text into sequences of discrete audio tokens have gained great research attention, with advances in neural audio…
Language ModelingLanguage ModellingQuantizationSpeech Synthesis+3Efficient Generative Modeling with Residual Vector Quantization-Based Tokens
We explore the use of Residual Vector Quantization (RVQ) for high-fidelity generation in vector-quantized generative models. This quantization technique maintains higher data fidelity by employing more in-depth tokens. H…
Conditional Image GenerationImage GenerationQuantizationSpeech Synthesis+4