Learning Disentangled Phone and Speaker Representations in a Semi-Supervised VQ-VAE Paradigm
We present a new approach to disentangle speaker voice and phone content by introducing new components to the VQ-VAE architecture for speech synthesis. The original VQ-VAE does not generalize well to unseen speakers or content. To alleviate this problem, we have incorporated a speaker encoder and speaker VQ codebook that learns global speaker characteristics entirely separate from the existing sub-phone codebooks. We also compare two training methods: self-supervised with global conditions and semi-supervised with speaker labels. Adding a speaker VQ component improves objective measures of speech synthesis quality (estimated MOS, speaker similarity, ASR-based intelligibility) and provides learned representations that are meaningful. Our speaker VQ codebook indices can be used in a simple speaker diarization task and perform slightly better than an x-vector baseline. Additionally, phones can be recognized from sub-phone VQ codebook indices in our semi-supervised VQ-VAE better than self-supervised with global conditions.
Code (1)
Tasks
speaker-diarizationSpeaker DiarizationSpeech SynthesisMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Exploring Disentanglement with Multilingual and Monolingual VQ-VAE
This work examines the content and usefulness of disentangled phone and speaker representations from two separately trained VQ-VAE systems: one trained on multilingual data and another trained on monolingual data. We exp…
DisentanglementBoosting Multi-Speaker Expressive Speech Synthesis with Semi-supervised Contrastive Learning
This paper aims to build a multi-speaker expressive TTS system, synthesizing a target speaker's speech with multiple styles and emotions. To this end, we propose a novel contrastive learning-based TTS approach to transfe…
Contrastive LearningExpressive Speech SynthesisSpeech SynthesisZero-shot text-to-speech synthesis conditioned using self-supervised speech representation model
This paper proposes a zero-shot text-to-speech (TTS) conditioned by a self-supervised speech-representation model acquired through self-supervised learning (SSL). Conventional methods with embedding vectors from x-vector…
RhythmSelf-Supervised LearningSpeech Synthesistext-to-speech+2Speaker-Disentangled Chunk-Wise Regression for Syllabic Tokenization
Unsupervised syllabic tokenization aims to learn discrete syllabic tokens that capture latent linguistic content-related structure from raw speech. Recent syllabic tokenization methods employ teacher-student distillation…
Boundary DetectionSemi-supervised learning for continuous emotional intensity controllable speech synthesis with disentangled representations
Recent text-to-speech models have reached the level of generating natural speech similar to what humans say. But there still have limitations in terms of expressiveness. The existing emotional speech synthesis models hav…
Emotional Speech SynthesisSpeech Synthesistext-to-speechText to Speech