Joint Training And Decoding for Multilingual End-to-End Simultaneous Speech Translation
Recent studies on end-to-end speech translation(ST) have facilitated the exploration of multilingual end-to-end ST and end-to-end simultaneous ST. In this paper, we investigate end-to-end simultaneous speech translation in a one-to-many multilingual setting which is closer to applications in real scenarios. We explore a separate decoder architecture and a unified architecture for joint synchronous training in this scenario. To further explore knowledge transfer across languages, we propose an asynchronous training strategy on the proposed unified decoder architecture. A multi-way aligned multilingual end-to-end ST dataset was curated as a benchmark testbed to evaluate our methods. Experimental results demonstrate the effectiveness of our models on the collected dataset. Our codes and data are available at: https://github.com/XiaoMi/TED-MMST.
Code (0)
등록된 구현이 없습니다.
Tasks
DecoderTransfer LearningTranslationSimilar Papers 제목 키워드 기반
Multilingual Approach to Joint Speech and Accent Recognition with DNN-HMM Framework
Human can recognize speech, as well as the peculiar accent of the speech simultaneously. However, present state-of-the-art ASR system can rarely do that. In this paper, we propose a multilingual approach to recognizing E…
speech-recognitionSpeech RecognitionTransfer LearningBi-directional Context-Enhanced Speech Large Language Models for Multilingual Conversational ASR
This paper introduces the integration of language-specific bi-directional context into a speech large language model (SLLM) to improve multilingual continuous conversational automatic speech recognition (ASR). We propose…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Language ModelingLanguage Modelling+3Multilingual Speech Recognition for Low-Resource Indian Languages using Multi-Task conformer
Transformers have recently become very popular for sequence-to-sequence applications such as machine translation and speech recognition. In this work, we propose a multi-task learning-based transformer model for low-reso…
DecoderMachine TranslationMulti-Task LearningPhoneme Recognition+3A Pre-trained Framework for Multilingual Brain Decoding Using Non-invasive Recordings
Brain-computer interfaces (BCIs) with speech decoding from brain recordings have broad application potential in fields such as clinical rehabilitation and cognitive neuroscience. However, current decoding methods remain …
Brain DecodingFairnessNAIST Simultaneous Speech Translation System for IWSLT 2024
This paper describes NAIST's submission to the simultaneous track of the IWSLT 2024 Evaluation Campaign: English-to-{German, Japanese, Chinese} speech-to-text translation and English-to-Japanese speech-to-speech translat…
Speech-to-Speech TranslationSpeech-to-TextSpeech-to-Text Translationtext-to-speech+2