CTC-based Compression for Direct Speech Translation
Previous studies demonstrated that a dynamic phone-informed compression of the input audio is beneficial for speech translation (ST). However, they required a dedicated model for phone recognition and did not test this solution for direct ST, in which a single model translates the input audio into the target language without intermediate representations. In this work, we propose the first method able to perform a dynamic compression of the input indirect ST models. In particular, we exploit the Connectionist Temporal Classification (CTC) to compress the input sequence according to its phonetic characteristics. Our experiments demonstrate that our solution brings a 1.3-1.5 BLEU improvement over a strong baseline on two language pairs (English-Italian and English-German), contextually reducing the memory footprint by more than 10%.
Code (1)
Tasks
TranslationSimilar Papers 제목 키워드 기반
Speechformer: Reducing Information Loss in Direct Speech Translation
Transformer-based models have gained increasing popularity achieving state-of-the-art performance in many research fields including speech translation. However, Transformer's quadratic complexity with respect to the inpu…
Speech-to-Text TranslationTranslationEfficient Speech Translation through Model Compression and Knowledge Distillation
Efficient deployment of large audio-language models for speech translation remains challenging due to their significant computational requirements. In this paper, we address this challenge through our system submissions …
Knowledge DistillationModel CompressionQuantizationTranslationPushing the Limits of Zero-shot End-to-End Speech Translation
Data scarcity and the modality gap between the speech and text modalities are two major obstacles of end-to-end Speech Translation (ST) systems, thus hindering their performance. Prior work has attempted to mitigate thes…
Speech-to-TextSpeech-to-Text TranslationTranslationDirect Simultaneous Speech-to-Speech Translation with Variational Monotonic Multihead Attention
We present a direct simultaneous speech-to-speech translation (Simul-S2ST) model, Furthermore, the generation of translation is independent from intermediate text representations. Our approach leverages recent progress o…
Simultaneous Speech-to-Speech TranslationSpeech SynthesisSpeech-to-Speech TranslationTranslationCascaded Models With Cyclic Feedback For Direct Speech Translation
Direct speech translation describes a scenario where only speech inputs and corresponding translations are available. Such data are notoriously limited. We present a technique that allows cascades of automatic speech rec…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Machine Translationspeech-recognition+2