DOCmT5: Document-Level Pretraining of Multilingual Language Models
In this paper, we introduce DOCmT5, a multilingual sequence-to-sequence language model pretrained with large scale parallel documents. While previous approaches have focused on leveraging sentence-level parallel data, we try to build a general-purpose pretrained model that can understand and generate long documents. We propose a simple and effective pretraining objective - Document reordering Machine Translation (DrMT), in which the input documents that are shuffled and masked need to be translated. DrMT brings consistent improvements over strong baselines on a variety of document-level generation tasks, including over 12 BLEU points for seen-language-pair document-level MT, over 7 BLEU points for unseen-language-pair document-level MT and over 3 ROUGE-1 points for seen-language-pair cross-lingual summarization. We achieve state-of-the-art (SOTA) on WMT20 De-En and IWSLT15 Zh-En document translation tasks. We also conduct extensive analysis on various factors for document pretraining, including (1) The effects of pretraining data quality and (2) The effects of combining mono-lingual and cross-lingual pretraining. We plan to make our model checkpoints publicly available.
Code (0)
등록된 구현이 없습니다.
Tasks
de-enDocument SummarizationDocument TranslationLanguage ModelingLanguage ModellingMachine TranslationSentenceTranslationSimilar Papers 제목 키워드 기반
DOCmT5: Document-Level Pre-training of Multilingual Language Models
In this paper, we introduce DOCmT5, a multi-lingual sequence-to-sequence language model pre-trained with large scale parallel documents. While previous approaches have focused on leveraging sentence-level parallel data, …
de-enDocument TranslationLanguage ModelingLanguage Modelling+1Fine-Grained and Multi-Dimensional Metrics for Document-Level Machine Translation
Large language models (LLMs) have excelled in various NLP tasks, including machine translation (MT), yet most studies focus on sentence-level translation. This work investigates the inherent capability of instruction-tun…
Document Level Machine TranslationMachine TranslationSentenceTranslationAdapting Large Language Models for Document-Level Machine Translation
Large language models (LLMs) have significantly advanced various natural language processing (NLP) tasks. Recent research indicates that moderately-sized LLMs often outperform larger ones after task-specific fine-tuning.…
Document Level Machine TranslationDomain GeneralizationMachine TranslationTranslationGRAFT: A Graph-based Flow-aware Agentic Framework for Document-level Machine Translation
Document level Machine Translation (DocMT) approaches often struggle with effectively capturing discourse level phenomena. Existing approaches rely on heuristic rules to segment documents into discourse units, which rare…
Document Level Machine TranslationDocument TranslationLarge Language ModelMachine Translation+1Doc-Guided Sent2Sent++: A Sent2Sent++ Agent with Doc-Guided memory for Document-level Machine Translation
The field of artificial intelligence has witnessed significant advancements in natural language processing, largely attributed to the capabilities of Large Language Models (LLMs). These models form the backbone of Agents…
Document Level Machine TranslationMachine TranslationSentenceTranslation