Exploring Phoneme-Level Speech Representations for End-to-End Speech Translation
Previous work on end-to-end translation from speech has primarily used frame-level features as speech representations, which creates longer, sparser sequences than text. We show that a naive method to create compressed phoneme-like speech representations is far more effective and efficient for translation than traditional frame-level speech features. Specifically, we generate phoneme labels for speech frames and average consecutive frames with the same label to create shorter, higher-level source sequences for translation. We see improvements of up to 5 BLEU on both our high and low resource language pairs, with a reduction in training time of 60%. Our improvements hold across multiple data sizes and two language pairs.
Code (0)
등록된 구현이 없습니다.
Tasks
TranslationSimilar Papers 제목 키워드 기반
Speech Representations and Phoneme Classification for Preserving the Endangered Language of Ladin
A vast majority of the world's 7,000 spoken languages are predicted to become extinct within this century, including the endangered language of Ladin from the Italian Alps. Linguists who work to preserve a language's pho…
DyPCL: Dynamic Phoneme-level Contrastive Learning for Dysarthric Speech Recognition
Dysarthric speech recognition often suffers from performance degradation due to the intrinsic diversity of dysarthric severity and extrinsic disparity from normal speech. To bridge these gaps, we propose a Dynamic Phonem…
Contrastive LearningDiversityspeech-recognitionSpeech RecognitionPredicting within and across language phoneme recognition performance of self-supervised learning speech pre-trained models
In this work, we analyzed and compared speech representations extracted from different frozen self-supervised learning (SSL) speech pre-trained models on their ability to capture articulatory features (AF) information an…
Phoneme RecognitionSelf-Supervised LearningEncoding of phonology in a recurrent neural model of grounded speech
We study the representation and encoding of phonemes in a recurrent neural network model of grounded speech. We use a model which processes images and their spoken descriptions, and projects the visual and auditory repre…
ClusteringOn granularity of prosodic representations in expressive text-to-speech
In expressive speech synthesis it is widely adopted to use latent prosody representations to deal with variability of the data during training. Same text may correspond to various acoustic realizations, which is known as…
Expressive Speech SynthesisSpeech Synthesistext-to-speechText to Speech