OpusTools and Parallel Corpus Diagnostics
This paper introduces OpusTools, a package for downloading and processing parallel corpora included in the OPUS corpus collection. The package implements tools for accessing compressed data in their archived release format and make it possible to easily convert between common formats. OpusTools also includes tools for language identification and data filtering as well as tools for importing data from various sources into the OPUS format. We show the use of these tools in parallel corpus creation and data diagnostics. The latter is especially useful for the identification of potential problems and errors in the extensive data set. Using these tools, we can now monitor the validity of data sets and improve the overall quality and consitency of the data collection.
Code (0)
등록된 구현이 없습니다.
Tasks
Language IdentificationSimilar Papers 제목 키워드 기반
The IIT Bombay English-Hindi Parallel Corpus
We present the IIT Bombay English-Hindi Parallel Corpus. The corpus is a compilation of parallel corpora previously available in the public domain as well as new parallel corpora we collected. The corpus contains 1.49 mi…
Machine TranslationNMTTranslationAlibaba Submission to the WMT18 Parallel Corpus Filtering Task
This paper describes the Alibaba Machine Translation Group submissions to the WMT 2018 Shared Task on Parallel Corpus Filtering. While evaluating the quality of the parallel corpus, the three characteristics of the corpu…
DiversityMachine TranslationSentenceTranslation+1Automatic Parallel Corpus Creation for Hindi-English News Translation Task
The parallel corpus for multilingual NLP tasks, deep learning applications like Statistical Machine Translation Systems is very important. The parallel corpus of Hindi-English language pair available for news translation…
Machine TranslationMultilingual NLPTranslationMachine Translation Model based on Non-parallel Corpus and Semi-supervised Transductive Learning
Although the parallel corpus has an irreplaceable role in machine translation, its scale and coverage is still beyond the actual needs. Non-parallel corpus resources on the web have an inestimable potential value in mach…
Machine TranslationTransductive LearningTranslationJParaCrawl v3.0: A Large-scale English-Japanese Parallel Corpus
Most current machine translation models are mainly trained with parallel corpora, and their translation accuracy largely depends on the quality and quantity of the corpora. Although there are billions of parallel sentenc…
Machine TranslationSentenceTranslation