Fine-grained Style Modeling, Transfer and Prediction in Text-to-Speech Synthesis via Phone-Level Content-Style Disentanglement
This paper presents a novel design of neural network system for fine-grained style modeling, transfer and prediction in expressive text-to-speech (TTS) synthesis. Fine-grained modeling is realized by extracting style embeddings from the mel-spectrograms of phone-level speech segments. Collaborative learning and adversarial learning strategies are applied in order to achieve effective disentanglement of content and style factors in speech and alleviate the "content leakage" problem in style modeling. The proposed system can be used for varying-content speech style transfer in the single-speaker scenario. The results of objective and subjective evaluation show that our system performs better than other fine-grained speech style transfer models, especially in the aspect of content preservation. By incorporating a style predictor, the proposed system can also be used for text-to-speech synthesis. Audio samples are provided for system demonstration https://daxintan-cuhk.github.io/pl-csd-speech .
Code (0)
등록된 구현이 없습니다.
Tasks
DisentanglementSpeech SynthesisStyle Transfertext-to-speechText to SpeechText-To-Speech SynthesisSimilar Papers 제목 키워드 기반
StylePTB: A Compositional Benchmark for Fine-grained Controllable Text Style Transfer
Text style transfer aims to controllably generate text with targeted stylistic changes while maintaining core meaning from the source sentence constant. Many of the existing style transfer benchmarks primarily focus on i…
BenchmarkingSentenceStyle TransferText Generation+1Improving Prosody for Cross-Speaker Style Transfer by Semi-Supervised Style Extractor and Hierarchical Modeling in Speech Synthesis
Cross-speaker style transfer in speech synthesis aims at transferring a style from source speaker to synthesized speech of a target speaker's timbre. In most previous methods, the synthesized fine-grained prosody feature…
Prosody PredictionSpeech SynthesisStyle TransferFine-Grained Image Style Transfer with Visual Transformers
With the development of the convolutional neural network, image style transfer has drawn increasing attention. However, most existing approaches adopt a global feature transformation to transfer style patterns into conte…
Style TransferOne-shot Embroidery Customization via Contrastive LoRA Modulation
Diffusion models have significantly advanced image manipulation techniques, and their ability to generate photorealistic images is beginning to transform retail workflows, particularly in presale visualization. Beyond ar…
Knowledge DistillationContrastive LearningImage ManipulationStyle TransferFine-Grained Emotion Prediction by Modeling Emotion Definitions
In this paper, we propose a new framework for fine-grained emotion prediction in the text through emotion definition modeling. Our approach involves a multi-task learning framework that models definitions of emotions as …
Language ModelingLanguage ModellingMasked Language ModelingMulti-Task Learning+2