Direct speech-to-speech translation with a sequence-to-sequence model
We present an attention-based sequence-to-sequence neural network which can directly translate speech from one language into speech in another language, without relying on an intermediate text representation. The network is trained end-to-end, learning to map speech spectrograms into target spectrograms in another language, corresponding to the translated content (in a different canonical voice). We further demonstrate the ability to synthesize translated speech using the voice of the source speaker. We conduct experiments on two Spanish-to-English speech translation datasets, and find that the proposed model slightly underperforms a baseline cascade of a direct speech-to-text translation model and a text-to-speech synthesis model, demonstrating the feasibility of the approach on this very challenging task.
Code (1)
Tasks
Speech SynthesisSpeech-to-Speech TranslationSpeech-to-TextSpeech-to-Text Translationtext-to-speechText to SpeechText-To-Speech SynthesisTranslationSimilar Papers 제목 키워드 기반
Sequence-to-Sequence Models Can Directly Translate Foreign Speech
We present a recurrent encoder-decoder deep neural network architecture that directly translates speech in one language into text in another. The model does not explicitly transcribe the speech into text in the source la…
DecoderMachine TranslationSequence-To-Sequence Speech Recognitionspeech-recognition+2STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translation
How to learn a better speech representation for end-to-end speech-to-text translation (ST) with limited labeled data? Existing techniques often attempt to transfer powerful machine translation (MT) capabilities to ST, bu…
Machine TranslationSpeech-to-TextSpeech-to-Text TranslationSpeech-to-speech Translation between Untranscribed Unknown Languages
In this paper, we explore a method for training speech-to-speech translation tasks without any transcription or linguistic supervision. Our proposed method consists of two steps: First, we train and generate discrete rep…
Speech-to-Speech TranslationTranslationDirect speech-to-speech translation with discrete units
We present a direct speech-to-speech translation (S2ST) model that translates speech from one language to speech in another language without relying on intermediate text generation. We tackle the problem by first applyin…
Speech-to-Speech TranslationText GenerationTranslationDuplex Diffusion Models Improve Speech-to-Speech Translation
Speech-to-speech translation is a typical sequence-to-sequence learning task that naturally has two directions. How to effectively leverage bidirectional supervision signals to produce high-fidelity audio for both direct…
Speech-to-Speech TranslationTranslation