Cross-Modal Multi-Tasking for Speech-to-Text Translation via Hard Parameter Sharing
Recent works in end-to-end speech-to-text translation (ST) have proposed multi-tasking methods with soft parameter sharing which leverage machine translation (MT) data via secondary encoders that map text inputs to an eventual cross-modal representation. In this work, we instead propose a ST/MT multi-tasking framework with hard parameter sharing in which all model parameters are shared cross-modally. Our method reduces the speech-text modality gap via a pre-processing stage which converts speech and text inputs into two discrete token sequences of similar length -- this allows models to indiscriminately process both modalities simply using a joint vocabulary. With experiments on MuST-C, we demonstrate that our multi-tasking framework improves attentional encoder-decoder, Connectionist Temporal Classification (CTC), transducer, and joint CTC/attention models by an average of +0.5 BLEU without any external MT data. Further, we show that this framework incorporates external MT data, yielding +0.8 BLEU, and also improves transfer learning from pre-trained textual models, yielding +1.8 BLEU.
Code (0)
등록된 구현이 없습니다.
Tasks
DecoderMachine TranslationSpeech-to-TextSpeech-to-Text TranslationTransfer LearningTranslationSimilar Papers 제목 키워드 기반
The University of Helsinki Submission to the IWSLT2020 Offline SpeechTranslation Task
This paper describes the University of Helsinki Language Technology group{'}s participation in the IWSLT 2020 offline speech translation task, addressing the translation of English audio into German text. In line with th…
Transfer LearningTranslationTowards Sentiment and Emotion aided Multi-modal Speech Act Classification in Twitter
Speech Act Classification determining the communicative intent of an utterance has been investigated widely over the years as a standalone task. This holds true for discussion in any fora including social media platform …
ClassificationEmotion RecognitionMAESTRO: Matched Speech Text Representations through Modality Matching
We present Maestro, a self-supervised training method to unify representations learnt from speech and text modalities. Self-supervised learning from speech signals aims to learn the latent structure inherent in the signa…
Language ModellingSelf-Supervised Learningspeech-recognitionSpeech Recognition+1Paralinguistics-Enhanced Large Language Modeling of Spoken Dialogue
Large Language Models (LLMs) have demonstrated superior abilities in tasks such as chatting, reasoning, and question-answering. However, standard LLMs may ignore crucial paralinguistic information, such as sentiment, emo…
AttributeLanguage ModelingLanguage ModellingQuestion Answering+3Bayesian Example Selection Improves In-Context Learning for Speech, Text, and Visual Modalities
Large language models (LLMs) can adapt to new tasks through in-context learning (ICL) based on a few examples presented in dialogue history without any model parameter update. Despite such convenience, the performance of…
In-Context Learning