Effectively pretraining a speech translation decoder with Machine Translation data
Directly translating from speech to text using an end-to-end approach is still challenging for many language pairs due to insufficient data. Although pretraining the encoder parameters using the Automatic Speech Recognition (ASR) task improves the results in low resource settings, attempting to use pretrained parameters from the Neural Machine Translation (NMT) task has been largely unsuccessful in previous works. In this paper, we will show that by using an adversarial regularizer, we can bring the encoder representations of the ASR and NMT tasks closer even though they are in different modalities, and how this helps us effectively use a pretrained NMT decoder for speech translation.
Code (0)
등록된 구현이 없습니다.
Tasks
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)DecoderMachine TranslationNMTspeech-recognitionSpeech RecognitionSpeech-to-TextTranslationSimilar Papers 제목 키워드 기반
Unveiling the Role of Pretraining in Direct Speech Translation
Direct speech-to-text translation systems encounter an important drawback in data scarcity. A common solution consists on pretraining the encoder on automatic speech recognition, hence losing efficiency in the training p…
Automatic Speech RecognitionDecoderspeech-recognitionSpeech Recognition+3Start-Before-End and End-to-End: Neural Speech Translation by AppTek and RWTH Aachen University
AppTek and RWTH Aachen University team together to participate in the offline and simultaneous speech translation tracks of IWSLT 2020. For the offline task, we create both cascaded and end-to-end speech translation syst…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)DecoderMachine Translation+4Compact Speech Translation Models via Discrete Speech Units Pretraining
We propose a pretraining method to use Self-Supervised Speech (SSS) model to creating more compact Speech-to-text Translation. In contrast to using the SSS model for initialization, our method is more suitable to memory …
DecoderSelf-Supervised LearningSpeech-to-TextSpeech-to-Text Translation+1Revisiting End-to-End Speech-to-Text Translation From Scratch
End-to-end (E2E) speech-to-text translation (ST) often depends on pretraining its encoder and/or decoder using source transcripts via speech recognition or text translation tasks, without which translation performance dr…
Decoderspeech-recognitionSpeech RecognitionSpeech-to-Text+2Pretraining by Backtranslation for End-to-end ASR in Low-Resource Settings
We explore training attention-based encoder-decoder ASR in low-resource settings. These models perform poorly when trained on small amounts of transcribed speech, in part because they depend on having sufficient target-s…
Data AugmentationDecoder