Self-supervised Context-aware Style Representation for Expressive Speech Synthesis
Expressive speech synthesis, like audiobook synthesis, is still challenging for style representation learning and prediction. Deriving from reference audio or predicting style tags from text requires a huge amount of labeled data, which is costly to acquire and difficult to define and annotate accurately. In this paper, we propose a novel framework for learning style representation from abundant plain text in a self-supervised manner. It leverages an emotion lexicon and uses contrastive learning and deep clustering. We further integrate the style representation as a conditioned embedding in a multi-style Transformer TTS. Comparing with multi-style TTS by predicting style tags trained on the same dataset but with human annotations, our method achieves improved results according to subjective evaluations on both in-domain and out-of-domain test sets in audiobook speech. Moreover, with implicit context-aware style representation, the emotion transition of synthesized audio in a long paragraph appears more natural. The audio samples are available on the demo web.
Code (0)
등록된 구현이 없습니다.
Tasks
Contrastive LearningDeep ClusteringExpressive Speech SynthesisRepresentation LearningSpeech SynthesisMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Speech-to-Speech Translation with Discrete-Unit-Based Style Transfer
Direct speech-to-speech translation (S2ST) with discrete self-supervised representations has achieved remarkable accuracy, but is unable to preserve the speaker timbre of the source speech. Meanwhile, the scarcity of hig…
In-Context LearningLanguage ModelingLanguage ModellingSpeech-to-Speech Translation+2SimAN: Exploring Self-Supervised Representation Learning of Scene Text via Similarity-Aware Normalization
Recently self-supervised representation learning has drawn considerable attention from the scene text recognition community. Different from previous studies using contrastive learning, we tackle the issue from an alterna…
Common Sense ReasoningContrastive LearningRepresentation LearningScene Text RecognitionTransductive Learning for Unsupervised Text Style Transfer
Unsupervised style transfer models are mainly based on an inductive learning approach, which represents the style as embeddings, decoder parameters, or discriminator parameters and directly applies these general rules to…
DecoderInductive LearningRetrievalStyle Transfer+3Contextual Text Style Transfer
We introduce a new task, Contextual Text Style Transfer - translating a sentence into a desired style with its surrounding context taken into account. This brings two key challenges to existing style transfer approaches:…
SentenceStyle TransferText Style TransferTranslationvq-wav2vec: Self-Supervised Learning of Discrete Speech Representations
We propose vq-wav2vec to learn discrete representations of audio segments through a wav2vec-style self-supervised context prediction task. The algorithm uses either a gumbel softmax or online k-means clustering to quanti…
ClusteringGeneral ClassificationSelf-Supervised Learningspeech-recognition+1