Benchmarking Generative Latent Variable Models for Speech
Stochastic latent variable models (LVMs) achieve state-of-the-art performance on natural image generation but are still inferior to deterministic models on speech. In this paper, we develop a speech benchmark of popular temporal LVMs and compare them against state-of-the-art deterministic models. We report the likelihood, which is a much used metric in the image domain, but rarely, or incomparably, reported for speech models. To assess the quality of the learned representations, we also compare their usefulness for phoneme recognition. Finally, we adapt the Clockwork VAE, a state-of-the-art temporal LVM for video generation, to the speech domain. Despite being autoregressive only in latent space, we find that the Clockwork VAE can outperform previous LVMs and reduce the gap to deterministic models by using a hierarchy of latent variables.
Code (1)
Tasks
BenchmarkingImage GenerationPhoneme RecognitionVideo GenerationSimilar Papers 제목 키워드 기반
Hierarchical Multi-Grained Generative Model for Expressive Speech Synthesis
This paper proposes a hierarchical generative model with a multi-grained latent variable to synthesize expressive speech. In recent years, fine-grained latent variables are introduced into the text-to-speech synthesis th…
Expressive Speech SynthesisSpeech Synthesistext-to-speechText to Speech+1Deep Variational Generative Models for Audio-visual Speech Separation
In this paper, we are interested in audio-visual speech separation given a single-channel audio recording as well as visual information (lips movements) associated with each speaker. We propose an unsupervised technique …
Speech SeparationA Recurrent Variational Autoencoder for Speech Enhancement
This paper presents a generative approach to speech enhancement based on a recurrent variational autoencoder (RVAE). The deep generative speech model is trained using clean speech signals only, and it is combined with a …
Speech EnhancementTowards Generative Latent Variable Models for Speech
While stochastic latent variable models (LVMs) now achieve state-of-the-art performance on natural image generation, they are still inferior to deterministic models on speech. On natural images, these models have been pa…
Image GenerationVideo GenerationAudio-visual speech enhancement with a deep Kalman filter generative model
Deep latent variable generative models based on variational autoencoder (VAE) have shown promising performance for audiovisual speech enhancement (AVSE). The underlying idea is to learn a VAEbased audiovisual prior distr…
Speech Enhancement