MultiUN v2: UN Documents with Multilingual Alignments
MultiUN is a multilingual parallel corpus extracted from the official documents of the United Nations. It is available in the six official languages of the UN and a small portion of it is also available in German. This paper presents a major update on the first public version of the corpus released in 2010. This version 2 consists of over 513,091 documents, including more than 9{\%} of new documents retrieved from the United Nations official document system. We applied several modifications to the corpus preparation method. In this paper, we describe the methods we used for processing the UN documents and aligning the sentences. The most significant improvement compared to the previous release is the newly added multilingual sentence alignment information. The alignment information is encoded together with the text in XML instead of additional files. Our representation of the sentence alignment allows quick construction of aligned texts parallel in arbitrary number of languages, which is essential for building machine translation systems.
Code (0)
등록된 구현이 없습니다.
Tasks
Information RetrievalMachine TranslationSentenceTranslationSimilar Papers 제목 키워드 기반
Do Explicit Alignments Robustly Improve Multilingual Encoders?
Multilingual BERT (mBERT), XLM-RoBERTa (XLMR) and other unsupervised multilingual encoders can effectively learn cross-lingual representation. Explicit alignment objectives based on bitexts like Europarl or MultiUN have …
The Role of Mixed-Language Documents for Multilingual Large Language Model Pretraining
Multilingual large language models achieve impressive cross-lingual performance despite largely monolingual pretraining. While bilingual data in pretraining corpora is widely believed to enable these abilities, details o…
Acquisitive Crimes, Time of Day, and Multiunit Housing in the City of Milwaukee
According to "Social Disorganization" theory, criminal activity increases if the societal institutions that might be responsible for maintaining order are weakened. Do large apartment buildings, which often have fairly t…
regressionOmniAlign: A Unified Multilingual Aligner for Word and Sentence Alignment
Cross-lingual sequence alignment is fundamental for building and exploiting parallel corpora, spanning mappings from documents and sentences down to words and subwords. Existing tools, however, typically specialize in a …
Long-Context UnderstandingSelf-Supervised LearningWord AlignmentScaling Unsupervised Word Alignment to Documents via Structural Constraints
Word alignment has traditionally been studied between sentences, but many cross-lingual tasks increasingly require correspondences across full documents. While recent multilingual embedding models can encode long inputs,…
Word Alignment