A multi-speaker multi-lingual voice cloning system based on vits2 for limmits 2024 challenge

2024-06-22 · Xiaopeng Wang, Yi Lu, Xin Qi, Zhiyong Wang, Yuankun Xie, Shuchen Shi, Ruibo Fu

This paper presents the development of a speech synthesis system for the LIMMITS'24 Challenge, focusing primarily on Track 2. The objective of the challenge is to establish a multi-speaker, multi-lingual Indic Text-to-Speech system with voice cloning capabilities, covering seven Indian languages with both male and female speakers. The system was trained using challenge data and fine-tuned for few-shot voice cloning on target speakers. Evaluation included both mono-lingual and cross-lingual synthesis across all seven languages, with subjective tests assessing naturalness and speaker similarity. Our system uses the VITS2 architecture, augmented with a multi-lingual ID and a BERT model to enhance contextual language comprehension. In Track 1, where no additional data usage was permitted, our model achieved a Speaker Similarity score of 4.02. In Track 2, which allowed the use of extra data, it attained a Speaker Similarity score of 4.17.

📄 PDF Abstract BibTeX arXiv:2406.17801

Code (0)

등록된 구현이 없습니다.

Tasks

Speech Synthesis text-to-speech Text to SpeechVoice Cloning

Methods 이 논문이 사용한 방법론

Refunds@Expedia|||How do I get a full refund from Expedia? “How do I get a full refund from Expedia? How do I get a full refund from Expedia? – Call ☎️ +1-(888) 829 (0881) or +1-805-330-4056 or +1-805-330-4056 for Quick Help &…

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Weight Decay 설명 없음

Residual Connection 설명 없음

Multi-Head Attention 설명 없음

WordPiece 설명 없음

Improve Cross-lingual Voice Cloning Using Low-quality Code-switched Data

2021-10-14 · Haitong Zhang, Yue Lin

Recently, sequence-to-sequence (seq-to-seq) models have been successfully applied in text-to-speech (TTS) to synthesize speech for single-language text. To synthesize speech for multiple languages usually requires multi-…

text-to-speech Text to Speech Voice Cloning

OpenVoice: Versatile Instant Voice Cloning

2023-12-03 · Zengyi Qin, Wenliang Zhao, Xumin Yu, Xin Sun

We introduce OpenVoice, a versatile voice cloning approach that requires only a short audio clip from the reference speaker to replicate their voice and generate speech in multiple languages. OpenVoice represents a signi…

Rhythm Voice Cloning

Your Voice Cloning System is Secretly a Voice Anonymizer

2026-08-27 · Romolo Muletta, Felix Matthias Saaro, Mark Cieliebak, Jan Deriu arxiv

Speaker anonymization suppresses speaker-identifying attributes from speech while preserving linguistic content and quality. We propose repurposing XTTSv2, a multilingual voice cloning model trained on 27k hours of speec…

Voice Conversion

One Voice, Many Tongues: Cross-Lingual Voice Cloning for Scientific Speech

2026-04-28 · Amanuel Gizachew Abebe, Yasmin Moslem arxiv

Preserving a speaker's voice identity while generating speech in a different language remains a fundamental challenge in spoken language technology, particularly in specialized domains such as scientific communication. I…

Data Augmentation

ERNIE-SAT: Speech and Text Joint Pretraining for Cross-Lingual Multi-Speaker Text-to-Speech

2022-11-07 · Xiaoran Fan, Chao Pang, Tian Yuan, He Bai 외

Speech representation learning has improved both speech understanding and speech synthesis tasks for single language. However, its ability in cross-lingual scenarios has not been explored. In this paper, we extend the pr…

Representation Learning Speech Representation Learning Speech Synthesis text-to-speech+3