T-Modules: Translation Modules for Zero-Shot Cross-Modal Machine Translation
We present a new approach to perform zero-shot cross-modal transfer between speech and text for translation tasks. Multilingual speech and text are encoded in a joint fixed-size representation space. Then, we compare different approaches to decode these multimodal and multilingual fixed-size representations, enabling zero-shot translation between languages and modalities. All our models are trained without the need of cross-modal labeled translation data. Despite a fixed-size representation, we achieve very competitive results on several text and speech translation tasks. In particular, we significantly improve the state-of-the-art for zero-shot speech translation on Must-C. Incorporating a speech decoder in our framework, we introduce the first results for zero-shot direct speech-to-speech and text-to-speech translation.
Code (0)
등록된 구현이 없습니다.
Tasks
DecoderMachine Translationtext-to-speechText to Speechtext-to-speech translationTranslationSimilar Papers 제목 키워드 기반
Revisiting Modularized Multilingual NMT to Meet Industrial Demands
The complete sharing of parameters for multilingual translation (1-1) has been the mainstream approach in current research. However, degraded performance due to the capacity bottleneck and low maintainability hinders its…
Machine TranslationNMTTranslationImproving Zero-shot Neural Machine Translation on Language-specific Encoders-Decoders
Recently, universal neural machine translation (NMT) with shared encoder-decoder gained good performance on zero-shot translation. Unlike universal NMT, jointly trained language-specific encoders-decoders aim to achieve …
DecoderDenoisingMachine TranslationNMT+1From Bilingual to Multilingual Neural Machine Translation by Incremental Training
Multilingual Neural Machine Translation approaches are based on the use of task-specific models and the addition of one more language can only be done by retraining the whole system. In this work, we propose a new traini…
DecoderMachine TranslationTranslationMultilingual Neural Machine Translation With the Right Amount of Sharing
Large multilingual Transformer-based machine translation models have had a pivotal role in making translation systems available for hundreds of languages with good zero-shot translation performance. One such example is t…
DecoderMachine TranslationNMTTranslationLanguage and Task Arithmetic with Parameter-Efficient Layers for Zero-Shot Summarization
Parameter-efficient fine-tuning (PEFT) using labeled task data can significantly improve the performance of large language models (LLMs) on the downstream task. However, there are 7000 languages in the world and many of …
Cross-Lingual Transferparameter-efficient fine-tuningTask ArithmeticText Generation+1