TMT: Tri-Modal Translation between Speech, Image, and Text by Processing Different Modalities as Different Languages
The capability to jointly process multi-modal information is becoming an essential task. However, the limited number of paired multi-modal data and the large computational requirements in multi-modal learning hinder the development. We propose a novel Tri-Modal Translation (TMT) model that translates between arbitrary modalities spanning speech, image, and text. We introduce a novel viewpoint, where we interpret different modalities as different languages, and treat multi-modal translation as a well-established machine translation problem. To this end, we tokenize speech and image data into discrete tokens, which provide a unified interface across modalities and significantly decrease the computational cost. In the proposed TMT, a multi-modal encoder-decoder conducts the core translation, whereas modality-specific processing is conducted only within the tokenization and detokenization stages. We evaluate the proposed TMT on all six modality translation tasks. TMT outperforms single model counterparts consistently, demonstrating that unifying tasks is beneficial not only for practicality but also for performance.
Code (1)
Tasks
DecoderMachine TranslationTranslationSimilar Papers 제목 키워드 기반
T-Modules: Translation Modules for Zero-Shot Cross-Modal Machine Translation
We present a new approach to perform zero-shot cross-modal transfer between speech and text for translation tasks. Multilingual speech and text are encoded in a joint fixed-size representation space. Then, we compare dif…
DecoderMachine Translationtext-to-speechText to Speech+2Scalable Multilingual Multimodal Machine Translation with Speech-Text Fusion
Multimodal Large Language Models (MLLMs) have achieved notable success in enhancing translation performance by integrating multimodal information. However, existing research primarily focuses on image-guided methods, who…
Multimodal Machine TranslationCMOT: Cross-modal Mixup via Optimal Transport for Speech Translation
End-to-end speech translation (ST) is the task of translating speech signals in the source language into text in the target language. As a cross-modal task, end-to-end ST is difficult to train with limited data. Existing…
Machine TranslationTranslationImproving speech translation by fusing speech and text
In speech translation, leveraging multimodal data to improve model performance and address limitations of individual modalities has shown significant effectiveness. In this paper, we harness the complementary strengths o…
cross-modal alignmentMachine TranslationTranslationM-Adapter: Modality Adaptation for End-to-End Speech-to-Text Translation
End-to-end speech-to-text translation models are often initialized with pre-trained speech encoder and pre-trained text decoder. This leads to a significant training gap between pre-training and fine-tuning, largely due …
DecoderSpeech-to-TextSpeech-to-Text TranslationTranslation