When End-to-End is Overkill: Rethinking Cascaded Speech-to-Text Translation

2025-02-01 · Anna Min, Chenxu Hu, Yi Ren, Hang Zhao

Though end-to-end speech-to-text translation has been a great success, we argue that the cascaded speech-to-text translation model still has its place, which is usually criticized for the error propagation between automatic speech recognition (ASR) and machine translation (MT) models. In this paper, we explore the benefits of incorporating multiple candidates from ASR and self-supervised speech features into MT. Our analysis reveals that the primary cause of cascading errors stems from the increased divergence between similar samples in the speech domain when mapped to the text domain. By including multiple candidates and self-supervised speech features, our approach allows the machine translation model to choose the right words and ensure precise translation using various speech samples. This strategy minimizes error spread and takes advantage of large ASR and MT datasets, along with pre-trained ASR/MT models, while addressing associated issues.

📄 PDF Abstract BibTeX arXiv:2502.00377

Code (0)

등록된 구현이 없습니다.

Tasks

Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Machine Translation speech-recognition Speech RecognitionSpeech-to-TextSpeech-to-Text Translation Translation

Speech Retrieval-Augmented Generation without Automatic Speech Recognition

2024-12-21 · Do June Min, Karel Mundnich, Andy Lapastora, Erfan Soltanmohammadi 외

One common approach for question answering over speech data is to first transcribe speech using automatic speech recognition (ASR) and then employ text-based retrieval-augmented generation (RAG) on the transcriptions. Wh…

Automatic Speech Recognition Automatic Speech Recognition (ASR)Language Modeling Language Modelling+7

SpeechCLIP+: Self-supervised multi-task representation learning for speech via CLIP and speech-image data

2024-02-10 · Hsuan-Fu Wang, Yi-Jen Shih, Heng-Jui Chang, Layne Berry 외

The recently proposed visually grounded speech model SpeechCLIP is an innovative framework that bridges speech and text through images via CLIP without relying on text transcription. On this basis, this paper introduces …

Keyword Extraction Multi-Task Learning Representation Learning Retrieval

Navigating the OverKill in Large Language Models

2024-01-31 · Chenyu Shi, Xiao Wang, Qiming Ge, Songyang Gao 외

Large language models are meticulously aligned to be both helpful and harmless. However, recent research points to a potential overkill which means models may refuse to answer benign queries. In this paper, we investigat…

Start-Before-End and End-to-End: Neural Speech Translation by AppTek and RWTH Aachen University

2020-07-01 · WS 2020 7 · Parnia Bahar, Patrick Wilken, Tamer Alkhouli, Andreas Guta 외

AppTek and RWTH Aachen University team together to participate in the offline and simultaneous speech translation tracks of IWSLT 2020. For the offline task, we create both cascaded and end-to-end speech translation syst…

Automatic Speech Recognition Automatic Speech Recognition (ASR)Decoder Machine Translation+4

KAME: Tandem Architecture for Enhancing Knowledge in Real-Time Speech-to-Speech Conversational AI

2025-09-26 · So Kuroki, Yotaro Kubo, Takuya Akiba, Yujin Tang arxiv

Real-time speech-to-speech (S2S) models excel at generating natural, low-latency conversational responses but often lack deep knowledge and semantic understanding. Conversely, cascaded systems combining automatic speech …

Text-To-Speech Synthesis Speech Recognition