Analyzing ASR pretraining for low-resource speech-to-text translation
Previous work has shown that for low-resource source languages, automatic speech-to-text translation (AST) can be improved by pretraining an end-to-end model on automatic speech recognition (ASR) data from a high-resource language. However, it is not clear what factors --e.g., language relatedness or size of the pretraining data-- yield the biggest improvements, or whether pretraining can be effectively combined with other methods such as data augmentation. Here, we experiment with pretraining on datasets of varying sizes, including languages related and unrelated to the AST source language. We find that the best predictor of final AST performance is the word error rate of the pretrained ASR model, and that differences in ASR/AST performance correlate with how phonetic information is encoded in the later RNN layers of our model. We also show that pretraining and data augmentation yield complementary benefits for AST.
Code (0)
등록된 구현이 없습니다.
Tasks
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Data Augmentationspeech-recognitionSpeech RecognitionSpeech-to-TextSpeech-to-Text TranslationTranslationSimilar Papers 제목 키워드 기반
Effectively pretraining a speech translation decoder with Machine Translation data
Directly translating from speech to text using an end-to-end approach is still challenging for many language pairs due to insufficient data. Although pretraining the encoder parameters using the Automatic Speech Recognit…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)DecoderMachine Translation+5Revisiting End-to-End Speech-to-Text Translation From Scratch
End-to-end (E2E) speech-to-text translation (ST) often depends on pretraining its encoder and/or decoder using source transcripts via speech recognition or text translation tasks, without which translation performance dr…
Decoderspeech-recognitionSpeech RecognitionSpeech-to-Text+2Pretraining by Backtranslation for End-to-end ASR in Low-Resource Settings
We explore training attention-based encoder-decoder ASR in low-resource settings. These models perform poorly when trained on small amounts of transcribed speech, in part because they depend on having sufficient target-s…
Data AugmentationDecoderThe Interpreter Understands Your Meaning: End-to-end Spoken Language Understanding Aided by Speech Translation
End-to-end spoken language understanding (SLU) remains elusive even with current large pretrained language models on text and speech, especially in multilingual cases. Machine translation has been established as a powerf…
Abstractive Text SummarizationContinual Learningintent-classificationIntent Classification+5Compact Speech Translation Models via Discrete Speech Units Pretraining
We propose a pretraining method to use Self-Supervised Speech (SSS) model to creating more compact Speech-to-text Translation. In contrast to using the SSS model for initialization, our method is more suitable to memory …
DecoderSelf-Supervised LearningSpeech-to-TextSpeech-to-Text Translation+1