Modular Speech-to-Text Translation for Zero-Shot Cross-Modal Transfer
Recent research has shown that independently trained encoders and decoders, combined through a shared fixed-size representation, can achieve competitive performance in speech-to-text translation. In this work, we show that this type of approach can be further improved with multilingual training. We observe significant improvements in zero-shot cross-modal speech translation, even outperforming a supervised approach based on XLSR for several languages.
Code (0)
등록된 구현이 없습니다.
Tasks
Speech-to-TextSpeech-to-Text TranslationTranslationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
T-Modules: Translation Modules for Zero-Shot Cross-Modal Machine Translation
We present a new approach to perform zero-shot cross-modal transfer between speech and text for translation tasks. Multilingual speech and text are encoded in a joint fixed-size representation space. Then, we compare dif…
DecoderMachine Translationtext-to-speechText to Speech+2Zero-shot Speech Translation
Speech Translation (ST) is the task of translating speech in one language into text in another language. Traditional cascaded approaches for ST, using Automatic Speech Recognition (ASR) and Machine Translation (MT) syste…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Machine Translationspeech-recognition+2Discrete Cross-Modal Alignment Enables Zero-Shot Speech Translation
End-to-end Speech Translation (ST) aims at translating the source language speech into target language text without generating the intermediate transcriptions. However, the training of end-to-end methods relies on parall…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)cross-modal alignmentMachine Translation+4RosettaSpeech: Zero-Shot Speech-to-Speech Translation without Parallel Speech
End-to-end speech-to-speech translation (S2ST) systems typically struggle with a critical data bottleneck: the scarcity of parallel speech-to-speech corpora. To overcome this, we introduce RosettaSpeech, a novel zero-sho…
Speech-to-Speech TranslationMachine TranslationTackling data scarcity in speech translation using zero-shot multilingual machine translation techniques
Recently, end-to-end speech translation (ST) has gained significant attention as it avoids error propagation. However, the approach suffers from data scarcity. It heavily depends on direct ST data and is less efficient i…
Data AugmentationMachine TranslationTranslation