홈 › Papers

RosettaSpeech: Zero-Shot Speech-to-Speech Translation without Parallel Speech

2025-11-26 · Zhisheng Zheng, Xiaohang Sun, Tuan Dinh, Abhishek Yanamandra, Abhinav Jain, Zhu Liu, Sunil Hadap, Vimal Bhat, Manoj Aggarwal, Gerard Medioni, David Harwath arxiv

End-to-end speech-to-speech translation (S2ST) systems typically struggle with a critical data bottleneck: the scarcity of parallel speech-to-speech corpora. To overcome this, we introduce RosettaSpeech, a novel zero-shot framework trained exclusively on monolingual speech-text data augmented by machine translation supervision. Unlike prior works that rely on complex cascaded pseudo-labeling, our approach strategically utilizes text as a semantic bridge during training to synthesize translation targets, thereby eliminating the need for parallel speech pairs while maintaining a direct, end-to-end inference pipeline. Empirical evaluations on the CVSS-C benchmark demonstrate that RosettaSpeech achieves state-of-the-art zero-shot performance, surpassing leading baselines by significant margins - achieving ASR-BLEU scores of 25.17 for German-to-English (+27% relative gain) and 29.86 for Spanish-to-English (+14%). Crucially, our model effectively preserves the source speaker's voice without ever seeing paired speech data. We further analyze the impact of data scaling and demonstrate the model's capability in many-to-one translation, offering a scalable solution for extending high-quality S2ST to "text-rich, speech-poor" languages.

📄 PDF Abstract BibTeX arXiv:2511.20974

Code (0)

등록된 구현이 없습니다.

Tasks

Speech-to-Speech Translation Machine Translation

Results from the Paper

Rank	Task	Dataset	Model	Metrics
#3	Speech-to-Speech Translation	CVSS	RosettaSpeech	ASR-BLEU: 25.17

T-Modules: Translation Modules for Zero-Shot Cross-Modal Machine Translation

2022-05-24 · Paul-Ambroise Duquenne, Hongyu Gong, Benoît Sagot, Holger Schwenk

We present a new approach to perform zero-shot cross-modal transfer between speech and text for translation tasks. Multilingual speech and text are encoded in a joint fixed-size representation space. Then, we compare dif…

Decoder Machine Translation text-to-speech Text to Speech+2

Discrete Cross-Modal Alignment Enables Zero-Shot Speech Translation

2022-10-18 · Chen Wang, Yuchen Liu, Boxing Chen, Jiajun Zhang 외

End-to-end Speech Translation (ST) aims at translating the source language speech into target language text without generating the intermediate transcriptions. However, the training of end-to-end methods relies on parall…

Automatic Speech Recognition Automatic Speech Recognition (ASR)cross-modal alignment Machine Translation+4

Zero-shot Speech Translation

2021-07-13 · Tu Anh Dinh

Speech Translation (ST) is the task of translating speech in one language into text in another language. Traditional cascaded approaches for ST, using Automatic Speech Recognition (ASR) and Machine Translation (MT) syste…

Automatic Speech Recognition Automatic Speech Recognition (ASR)Machine Translation speech-recognition+2

Modular Speech-to-Text Translation for Zero-Shot Cross-Modal Transfer

2023-10-05 · Paul-Ambroise Duquenne, Holger Schwenk, Benoît Sagot

Recent research has shown that independently trained encoders and decoders, combined through a shared fixed-size representation, can achieve competitive performance in speech-to-text translation. In this work, we show th…

Speech-to-Text Speech-to-Text Translation Translation

Pushing the Limits of Zero-shot End-to-End Speech Translation

2024-02-16 · Ioannis Tsiamas, Gerard I. Gállego, José A. R. Fonollosa, Marta R. Costa-jussà

Data scarcity and the modality gap between the speech and text modalities are two major obstacles of end-to-end Speech Translation (ST) systems, thus hindering their performance. Prior work has attempted to mitigate thes…