Soft Alignment of Modality Space for End-to-end Speech Translation
End-to-end Speech Translation (ST) aims to convert speech into target text within a unified model. The inherent differences between speech and text modalities often impede effective cross-modal and cross-lingual transfer. Existing methods typically employ hard alignment (H-Align) of individual speech and text segments, which can degrade textual representations. To address this, we introduce Soft Alignment (S-Align), using adversarial training to align the representation spaces of both modalities. S-Align creates a modality-invariant space while preserving individual modality quality. Experiments on three languages from the MuST-C dataset show S-Align outperforms H-Align across multiple tasks and offers translation capabilities on par with specialized translation models.
Code (0)
등록된 구현이 없습니다.
Tasks
Cross-Lingual TransferTranslationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
CMOT: Cross-modal Mixup via Optimal Transport for Speech Translation
End-to-end speech translation (ST) is the task of translating speech signals in the source language into text in the target language. As a cross-modal task, end-to-end ST is difficult to train with limited data. Existing…
Machine TranslationTranslationTransferable speech-to-text large language model alignment module
By leveraging the power of Large Language Models(LLMs) and speech foundation models, state of the art speech-text bimodal works can achieve challenging tasks like spoken translation(ST) and question answering(SQA) altoge…
Language ModelingLanguage ModellingLarge Language ModelQuestion Answering+1DTW-Align: Bridging the Modality Gap in End-to-End Speech Translation with Dynamic Time Warping Alignment
End-to-End Speech Translation (E2E-ST) is the task of translating source speech directly into target text bypassing the intermediate transcription step. The representation discrepancy between the speech and text modaliti…
Adaptive Inner Speech-Text Alignment for LLM-based Speech Translation
Recent advancement of large language models (LLMs) has led to significant breakthroughs across various tasks, laying the foundation for the development of LLM-based speech translation systems. Existing methods primarily …
Cross-Modal RetrievalTranslationBLSP: Bootstrapping Language-Speech Pre-training via Behavior Alignment of Continuation Writing
The emergence of large language models (LLMs) has sparked significant interest in extending their remarkable language capabilities to speech. However, modality alignment between speech and text still remains an open prob…
speech-recognitionSpeech RecognitionSpoken Language Understanding