Joint training framework for text-to-speech and voice conversion using multi-source Tacotron and WaveNet
We investigated the training of a shared model for both text-to-speech (TTS) and voice conversion (VC) tasks. We propose using an extended model architecture of Tacotron, that is a multi-source sequence-to-sequence model with a dual attention mechanism as the shared model for both the TTS and VC tasks. This model can accomplish these two different tasks respectively according to the type of input. An end-to-end speech synthesis task is conducted when the model is given text as the input while a sequence-to-sequence voice conversion task is conducted when it is given the speech of a source speaker as the input. Waveform signals are generated by using WaveNet, which is conditioned by using a predicted mel-spectrogram. We propose jointly training a shared model as a decoder for a target speaker that supports multiple sources. Listening experiments show that our proposed multi-source encoder-decoder model can efficiently achieve both the TTS and VC tasks.
Code (0)
등록된 구현이 없습니다.
Tasks
DecoderSpeech Synthesistext-to-speechText to SpeechVoice ConversionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
ERNIE-SAT: Speech and Text Joint Pretraining for Cross-Lingual Multi-Speaker Text-to-Speech
Speech representation learning has improved both speech understanding and speech synthesis tasks for single language. However, its ability in cross-lingual scenarios has not been explored. In this paper, we extend the pr…
Representation LearningSpeech Representation LearningSpeech Synthesistext-to-speech+3DiffVoice: Text-to-Speech with Latent Diffusion
In this work, we present DiffVoice, a novel text-to-speech model based on latent diffusion. We propose to first encode speech signals into a phoneme-rate latent representation with a variational autoencoder enhanced by a…
text-to-speechText to SpeechUsing joint training speaker encoder with consistency loss to achieve cross-lingual voice conversion and expressive voice conversion
Voice conversion systems have made significant advancements in terms of naturalness and similarity in common voice conversion tasks. However, their performance in more complex tasks such as cross-lingual voice conversion…
speech-recognitionSpeech RecognitionVoice ConversionVevo2: A Unified and Controllable Framework for Speech and Singing Voice Generation
Controllable human voice generation, particularly for expressive domains like singing, remains a significant challenge. This paper introduces Vevo2, a unified framework for controllable speech and singing voice generatio…
VoiceBridge: General Speech Restoration with One-step Latent Bridge Models
Bridge models have been investigated in speech enhancement but are mostly single-task, with constrained general speech restoration (GSR) capability. In this work, we propose VoiceBridge, a one-step latent bridge model (L…
Speech Enhancement