Bifixer and Bicleaner: two open-source tools to clean your parallel data
This paper shows the utility of two open-source tools designed for parallel data cleaning: Bifixer and Bicleaner. Already used to clean highly noisy parallel content from crawled multilingual websites, we evaluate their performance in a different scenario: cleaning publicly available corpora commonly used to train machine translation systems. We choose four English–Portuguese corpora which we plan to use internally to compute paraphrases at a later stage. We clean the four corpora using both tools, which are described in detail, and analyse the effect of some of the cleaning steps on them. We then compare machine translation training times and quality before and after cleaning these corpora, showing a positive impact particularly for the noisiest ones.
Code (1)
Tasks
Machine TranslationTranslationSimilar Papers 제목 키워드 기반
Bicleaner at WMT 2020: Universitat d’Alacant-Prompsit’s submission to the parallel corpus filtering shared task
This paper describes the joint submission of Universitat d’Alacant and Prompsit Language Engineering to the WMT 2020 shared task on parallel corpus filtering. Our submission, based on the free/open-source tool Bicleaner,…
Bicleaner AI: Bicleaner Goes Neural
This paper describes the experiments carried out during the development of the latest version of Bicleaner, named Bicleaner AI, a tool that aims at detecting noisy sentences in parallel corpora. The tool, which now imple…
Binary ClassificationMachine TranslationNMTTranslationCleanRL: High-quality Single-file Implementations of Deep Reinforcement Learning Algorithms
CleanRL is an open-source library that provides high-quality single-file implementations of Deep Reinforcement Learning algorithms. It provides a simpler yet scalable developing experience by having a straightforward cod…
BenchmarkingDeep Reinforcement Learningreinforcement-learningReinforcement Learning (RL)+1OpusCleaner and OpusTrainer, open source toolkits for training Machine Translation and Large language models
Developing high quality machine translation systems is a labour intensive, challenging and confusing process for newcomers to the field. We present a pair of tools OpusCleaner and OpusTrainer that aim to simplify the pro…
Data AugmentationMachine TranslationSchedulingTranslationSelecting the best data filtering method for NMT training
Performance of NMT systems has been proven to depend on the quality of the training data. In this paper we explore different open-source tools that can be used to score the quality of translation pairs, with the goal of …
NMTTranslation