Towards Unsupervised Speech-to-Text Translation
We present a framework for building speech-to-text translation (ST) systems using only monolingual speech and text corpora, in other words, speech utterances from a source language and independent text from a target language. As opposed to traditional cascaded systems and end-to-end architectures, our system does not require any labeled data (i.e., transcribed source audio or parallel source and target text corpora) during training, making it especially applicable to language pairs with very few or even zero bilingual resources. The framework initializes the ST system with a cross-modal bilingual dictionary inferred from the monolingual corpora, that maps every source speech segment corresponding to a spoken word to its target text translation. For unseen source speech utterances, the system first performs word-by-word translation on each speech segment in the utterance. The translation is improved by leveraging a language model and a sequence denoising autoencoder to provide prior knowledge about the target language. Experimental results show that our unsupervised system achieves comparable BLEU scores to supervised end-to-end models despite the lack of supervision. We also provide an ablation analysis to examine the utility of each component in our system.
Code (0)
등록된 구현이 없습니다.
Tasks
DenoisingLanguage ModelingLanguage ModellingSpeech-to-TextSpeech-to-Text TranslationTranslationWord TranslationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Simple and Effective Unsupervised Speech Translation
The amount of labeled data to train models for speech tasks is limited for most languages, however, the data scarcity is exacerbated for speech translation which requires labeled data covering two different languages. To…
Domain AdaptationMachine Translationspeech-recognitionSpeech Recognition+6Unsupervised Cross-Modal Alignment of Speech and Text Embedding Spaces
Recent research has shown that word embedding spaces learned from text corpora of different languages can be aligned without any parallel data supervision. Inspired by the success in unsupervised cross-lingual word embed…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Cross-Lingual Word Embeddingscross-modal alignment+6Translatotron 3: Speech to Speech Translation with Monolingual Data
This paper presents Translatotron 3, a novel approach to unsupervised direct speech-to-speech translation from monolingual speech-text datasets by combining masked autoencoder, unsupervised embedding mapping, and back-tr…
Speech-to-Speech TranslationTranslationTowards speech-to-text translation without speech recognition
We explore the problem of translating speech to text in low-resource scenarios where neither automatic speech recognition (ASR) nor machine translation (MT) are available, but we have training data in the form of audio p…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Machine Translationspeech-recognition+4Spoken Term Discovery for Language Documentation using Translations
Vast amounts of speech data collected for language documentation and research remain untranscribed and unsearchable, but often a small amount of speech may have text translations available. We present a method for partia…
Translation