StreamAtt: Direct Streaming Speech-to-Text Translation with Attention-based Audio History Selection
Streaming speech-to-text translation (StreamST) is the task of automatically translating speech while incrementally receiving an audio stream. Unlike simultaneous ST (SimulST), which deals with pre-segmented speech, StreamST faces the challenges of handling continuous and unbounded audio streams. This requires additional decisions about what to retain of the previous history, which is impractical to keep entirely due to latency and computational constraints. Despite the real-world demand for real-time ST, research on streaming translation remains limited, with existing works solely focusing on SimulST. To fill this gap, we introduce StreamAtt, the first StreamST policy, and propose StreamLAAL, the first StreamST latency metric designed to be comparable with existing metrics for SimulST. Extensive experiments across all 8 languages of MuST-C v1.0 show the effectiveness of StreamAtt compared to a naive streaming baseline and the related state-of-the-art SimulST policy, providing a first step in StreamST research.
Code (1)
Tasks
Speech-to-TextSpeech-to-Text TranslationTranslationSimilar Papers 제목 키워드 기반
Textless Streaming Speech-to-Speech Translation using Semantic Speech Tokens
Cascaded speech-to-speech translation systems often suffer from the error accumulation problem and high latency, which is a result of cascaded modules whose inference delays accumulate. In this paper, we propose a transd…
Language ModelingLanguage ModellingMachine TranslationSpeech-to-Speech Translation+3Direct Simultaneous Speech-to-Text Translation Assisted by Synchronized Streaming ASR
Simultaneous speech-to-text translation is widely useful in many scenarios. The conventional cascaded approach uses a pipeline of streaming ASR followed by simultaneous MT, but suffers from error propagation and extra la…
Simultaneous Speech-to-Text TranslationSpeech-to-TextSpeech-to-Text TranslationTranslationLarge-Scale Streaming End-to-End Speech Translation with Neural Transducers
Neural transducers have been widely used in automatic speech recognition (ASR). In this paper, we introduce it to streaming end-to-end speech translation (ST), which aims to convert audio signals to texts in other langua…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Machine Translationspeech-recognition+2Streaming Speech-to-Text Translation with a SpeechLLM
Normally, a system that translates speech into text consists of separate modules for speech recognition and text-to-text translation. Combining those tasks into a SpeechLLM promises to exploit paralinguistic information …
Speech-to-Text TranslationSpeech RecognitionSystem Description on Automatic Simultaneous Translation Workshop
This paper describes our system submitted on the third automatic simultaneous translation workshop at NAACL2022. We participate in the Chinese audio->English text direction of Chinese-to-English translation. Our speech-t…
SentenceSpeech-to-TextTranslation