Improving Speech-to-Speech Translation Through Unlabeled Text
Direct speech-to-speech translation (S2ST) is among the most challenging problems in the translation paradigm due to the significant scarcity of S2ST data. While effort has been made to increase the data size from unlabeled speech by cascading pretrained speech recognition (ASR), machine translation (MT) and text-to-speech (TTS) models; unlabeled text has remained relatively under-utilized to improve S2ST. We propose an effective way to utilize the massive existing unlabeled text from different languages to create a large amount of S2ST data to improve S2ST performance by applying various acoustic effects to the generated synthetic data. Empirically our method outperforms the state of the art in Spanish-English translation by up to 2 BLEU. Significant gains by the proposed method are demonstrated in extremely low-resource settings for both Spanish-English and Russian-English translations.
Code (0)
등록된 구현이 없습니다.
Tasks
Machine Translationspeech-recognitionSpeech RecognitionSpeech-to-Speech Translationtext-to-speechText to SpeechTranslationSimilar Papers 제목 키워드 기반
Mu$^{2}$SLAM: Multitask, Multilingual Speech and Language Models
We present Mu$^{2}$SLAM, a multilingual sequence-to-sequence model pre-trained jointly on unlabeled speech, unlabeled text and supervised data spanning Automatic Speech Recognition (ASR), Automatic Speech Translation (AS…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)automatic-speech-translationDecoder+8Improving End-to-end Speech Translation by Leveraging Auxiliary Speech and Text Data
We present a method for introducing a text encoder into pre-training end-to-end speech translation systems. It enhances the ability of adapting one modality (i.e., source-language speech) to another (i.e., source-languag…
de-enDenoisingTranslationImproving End-to-end Speech Translation by Leveraging Auxiliary Speech and Text Data
We present a method for introducing a text encoder into pre-trained end-to-end speech translation systems. It enhances the ability of adapting one modality (i.e., source-language speech) to another (i.e., source-language…
de-enDenoisingTranslationSLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Training
Unsupervised pre-training is now the predominant approach for both text and speech understanding. Self-attention models pre-trained on large amounts of unannotated data have been hugely successful when fine-tuned on down…
Language ModelingLanguage ModellingText MatchingText Normalization+2mSLAM: Massively multilingual joint pre-training for speech and text
We present mSLAM, a multilingual Speech and LAnguage Model that learns cross-lingual cross-modal representations of speech and text by pre-training jointly on large amounts of unlabeled speech and text in multiple langua…
cross-modal alignmentintent-classificationIntent ClassificationLanguage Modeling+4