CrossVoice: Crosslingual Prosody Preserving Cascade-S2ST using Transfer Learning
This paper presents CrossVoice, a novel cascade-based Speech-to-Speech Translation (S2ST) system employing advanced ASR, MT, and TTS technologies with cross-lingual prosody preservation through transfer learning. We conducted comprehensive experiments comparing CrossVoice with direct-S2ST systems, showing improved BLEU scores on tasks such as Fisher Es-En, VoxPopuli Fr-En and prosody preservation on benchmark datasets CVSS-T and IndicTTS. With an average mean opinion score of 3.75 out of 4, speech synthesized by CrossVoice closely rivals human speech on the benchmark, highlighting the efficacy of cascade-based systems and transfer learning in multilingual S2ST with prosody transfer.
Code (0)
등록된 구현이 없습니다.
Tasks
es-enfr-enSpeech-to-Speech TranslationTransfer LearningTranslationSimilar Papers 제목 키워드 기반
A unified one-shot prosody and speaker conversion system with self-supervised discrete speech units
We present a unified system to realize one-shot voice conversion (VC) on the pitch, rhythm, and speaker attributes. Existing works generally ignore the correlation between prosody and language content, leading to the deg…
RhythmVoice ConversionExpressive Machine Dubbing Through Phrase-level Cross-lingual Prosody Transfer
Speech generation for machine dubbing adds complexity to conventional Text-To-Speech solutions as the generated output is required to match the expressiveness, emotion and speaking rate of the source content. Capturing a…
text-to-speechText to SpeechDirect Speech to Speech Translation: A Review
Speech to speech translation (S2ST) is a transformative technology that bridges global communication gaps, enabling real time multilingual interactions in diplomacy, tourism, and international trade. Our review examines …
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Machine Translationspeech-recognition+5A Neural TTS System with Parallel Prosody Transfer from Unseen Speakers
Modern neural TTS systems are capable of generating natural and expressive speech when provided with sufficient amounts of training data. Such systems can be equipped with prosody-control functionality, allowing for more…
CopyCat: Many-to-Many Fine-Grained Prosody Transfer for Neural Text-to-Speech
Prosody Transfer (PT) is a technique that aims to use the prosody from a source audio as a reference while synthesising speech. Fine-grained PT aims at capturing prosodic aspects like rhythm, emphasis, melody, duration, …
Rhythmtext-to-speechText to Speech