Selective Classifier-free Guidance for Zero-shot Text-to-speech

2025-09-24 · John Zheng, Farhad Maleki arxiv

In zero-shot text-to-speech, achieving a balance between fidelity to the target speaker and adherence to text content remains a challenge. While classifier-free guidance (CFG) strategies have shown promising results in image generation, their application to speech synthesis are underexplored. Separating the conditions used for CFG enables trade-offs between different desired characteristics in speech synthesis. In this paper, we evaluate the adaptability of CFG strategies originally developed for image generation to speech synthesis and extend separated-condition CFG approaches for this domain. Our results show that CFG strategies effective in image generation generally fail to improve speech synthesis. We also find that we can improve speaker similarity while limiting degradation of text adherence by applying standard CFG during early timesteps and switching to selective CFG only in later timesteps. Surprisingly, we observe that the effectiveness of a selective CFG strategy is highly text-representation dependent, as differences between the two languages of English and Mandarin can lead to different results even with the same model.

📄 PDF Abstract BibTeX arXiv:2509.19668

Code (0)

등록된 구현이 없습니다.

Tasks

Speech Synthesis Image Generation

Meta-Learning via Classifier(-free) Diffusion Guidance

2022-10-17 · Elvis Nava, Seijin Kobayashi, Yifei Yin, Robert K. Katzschmann 외

We introduce meta-learning algorithms that perform zero-shot weight-space adaptation of neural network models to unseen tasks. Our methods repurpose the popular generative image synthesis techniques of natural language g…

Few-Shot Learning Image Generation Meta-Learning Visual Question Answering (VQA)+1

Selective Zero-Shot Classification with Augmented Attributes

2018-07-19 · ECCV 2018 9 · Jie Song, Chengchao Shen, Jie Lei, An-Xiang Zeng 외

In this paper, we introduce a selective zero-shot classification problem: how can the classifier avoid making dubious predictions? Existing attribute-based zero-shot classification methods are shown to work poorly in the…

Attribute Classification General Classification zero-shot-classification+1

Understanding and Improving Training-free Loss-based Diffusion Guidance

2024-03-19 · Yifei Shen, Xinyang Jiang, Yezhen Wang, Yifan Yang 외

Adding additional control to pretrained diffusion models has become an increasingly popular research area, with extensive applications in computer vision, reinforcement learning, and AI for science. Recently, several stu…

Motion Generation

MaskVCT: Masked Voice Codec Transformer for Zero-Shot Voice Conversion With Increased Controllability via Multiple Guidances

2025-09-21 · Junhyeok Lee, Helin Wang, Yaohan Guan, Thomas Thebaud 외 arxiv

We introduce MaskVCT, a zero-shot voice conversion (VC) model that offers multi-factor controllability through multiple classifier-free guidances (CFGs). While previous VC models rely on a fixed conditioning scheme, Mask…

Voice Conversion

CuteTTS: Efficient and High-Quality Speech Synthesis via Autoregressive Modeling of Continuous Latents

2026-08-09 · Yuqian Zhang, Yao Shi, Kexin Huang, Botian Jiang 외 arxiv

Zero-shot text-to-speech (TTS) now supports interactive assistants, personalized media, and accessibility tools. All TTS systems require faithful linguistic rendering, consistent speaker identity, and low-latency respons…

Speech Synthesis