FASST: Fast LLM-based Simultaneous Speech Translation
Simultaneous speech translation (SST) takes streaming speech input and generates text translation on the fly. Existing methods either have high latency due to recomputation of input representations, or fall behind of offline ST in translation quality. In this paper, we propose FASST, a fast large language model based method for streaming speech translation. We propose blockwise-causal speech encoding and consistency mask, so that streaming speech input can be encoded incrementally without recomputation. Furthermore, we develop a two-stage training strategy to optimize FASST for simultaneous inference. We evaluate FASST and multiple strong prior models on MuST-C dataset. Experiment results show that FASST achieves the best quality-latency trade-off. It outperforms the previous best model by an average of 1.5 BLEU under the same latency for English to Spanish translation.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage ModellingLarge Language ModelTranslationSimilar Papers 제목 키워드 기반
Transcribing and Translating, Fast and Slow: Joint Speech Translation and Recognition
We propose the joint speech translation and recognition (JSTAR) model that leverages the fast-slow cascaded encoder architecture for simultaneous end-to-end automatic speech recognition (ASR) and speech translation (ST).…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Machine Translationspeech-recognition+2RASST: Retrieval-Augmented Simultaneous Speech Translation
Simultaneous speech translation produces target text incrementally from partial speech input. Recent speech large language models have markedly improved SST quality but still struggle with rare and domain-specific termin…
Cross-Modal RetrievalMachine TranslationSpeech RecognitionFluent and Low-latency Simultaneous Speech-to-Speech Translation with Self-adaptive Training
Simultaneous speech-to-speech translation is widely useful but extremely challenging, since it needs to generate target-language speech concurrently with the source-language speech, with only a few seconds delay. In addi…
SentenceSimultaneous Speech-to-Speech TranslationSpeech-to-Speech TranslationTranslationCUNI-KIT System for Simultaneous Speech Translation Task at IWSLT 2022
In this paper, we describe our submission to the Simultaneous Speech Translation at IWSLT 2022. We explore strategies to utilize an offline model in a simultaneous setting without the need to modify the original model. I…
TranslationStreamSpeech: Simultaneous Speech-to-Speech Translation with Multi-task Learning
Simultaneous speech-to-speech translation (Simul-S2ST, a.k.a streaming speech translation) outputs target speech while receiving streaming speech inputs, which is critical for real-time communication. Beyond accomplishin…
Automatic Speech Recognition (ASR)de-enes-enfr-en+11