PARADISE: Exploiting Parallel Data for Multilingual Sequence-to-Sequence Pretraining
Despite the success of multilingual sequence-to-sequence pretraining, most existing approaches rely on monolingual corpora, and do not make use of the strong cross-lingual signal contained in parallel data. In this paper, we present PARADISE (PARAllel & Denoising Integration in SEquence-to-sequence models), which extends the conventional denoising objective used to train these models by (i) replacing words in the noised sequence according to a multilingual dictionary, and (ii) predicting the reference translation according to a parallel corpus instead of recovering the original sequence. Our experiments on machine translation and cross-lingual natural language inference show an average improvement of 2.0 BLEU points and 6.7 accuracy points from integrating parallel data into pretraining, respectively, obtaining results that are competitive with several popular models at a fraction of their computational cost.
Code (1)
Tasks
Cross-Lingual Natural Language InferenceDenoisingMachine TranslationNatural Language InferenceTranslationSimilar Papers 제목 키워드 기반
PARADISE”:" Exploiting Parallel Data for Multilingual Sequence-to-Sequence Pretraining
Despite the success of multilingual sequence-to-sequence pretraining, most existing approaches rely on monolingual corpora and do not make use of the strong cross-lingual signal contained in parallel data. In this paper,…
Cross-Lingual Natural Language InferenceDenoisingMachine TranslationNatural Language Inference+1The Parallel Distributed Image Search Engine (ParaDISE)
Image retrieval is a complex task that differs according to the context and the user requirements in any specific field, for example in a medical environment. Search by text is often not possible or optimal and retrieval…
Image RetrievalMedical Image RetrievalRetrievalExploiting Domain-Specific Parallel Data on Multilingual Language Models for Low-resource Language Translation
Neural Machine Translation (NMT) systems built on multilingual sequence-to-sequence Language Models (msLMs) fail to deliver expected results when the amount of parallel data for a language, as well as the language's repr…
Machine TranslationNMTA Recipe of Parallel Corpora Exploitation for Multilingual Large Language Models
Recent studies have highlighted the potential of exploiting parallel corpora to enhance multilingual large language models, improving performance in both bilingual tasks, e.g., machine translation, and general-purpose ta…
Language IdentificationMachine TranslationSentencetext-classification+2Paved2Paradise: Cost-Effective and Scalable LiDAR Simulation by Factoring the Real World
To achieve strong real world performance, neural networks must be trained on large, diverse datasets; however, obtaining and annotating such datasets is costly and time-consuming, particularly for 3D point clouds. In thi…
Human DetectionPedestrian Detection