The Speechmatics Parallel Corpus Filtering System for WMT18
Our entry to the parallel corpus filtering task uses a two-step strategy. The first step uses a series of pragmatic hard {`}rules{'} to remove the worst example sentences. This first step reduces the effective corpus size down from the initial 1 billion to 160 million tokens. The second step uses four different heuristics weighted to produce a score that is then used for further filtering down to 100 or 10 million tokens. Our final system produces competitive results without requiring excessive fine tuning to the exact task or language pair. The first step in isolation provides a very fast filter that gives most of the gains of the final system.
Code (0)
등록된 구현이 없습니다.
Tasks
Machine TranslationSimilar Papers 제목 키워드 기반
Alibaba Submission to the WMT18 Parallel Corpus Filtering Task
This paper describes the Alibaba Machine Translation Group submissions to the WMT 2018 Shared Task on Parallel Corpus Filtering. While evaluating the quality of the parallel corpus, the three characteristics of the corpu…
DiversityMachine TranslationSentenceTranslation+1Accurate semantic textual similarity for cleaning noisy parallel corpora using semantic machine translation evaluation metric: The NRC supervised submissions to the Parallel Corpus Filtering task
We present our semantic textual similarity approach in filtering a noisy web crawled parallel corpus using YiSi{---}a novel semantic machine translation evaluation metric. The systems mainly based on this supervised appr…
Machine TranslationSemantic Textual SimilarityTranslationCoverage and Cynicism: The AFRL Submission to the WMT 2018 Parallel Corpus Filtering Task
The WMT 2018 Parallel Corpus Filtering Task aims to test various methods of filtering a noisy parallel corpus, to make it useful for training machine translation systems. We describe the AFRL submissions, including their…
Machine TranslationTranslationTilde's Parallel Corpus Filtering Methods for WMT 2018
The paper describes parallel corpus filtering methods that allow reducing noise of noisy {``}parallel{''} corpora from a level where the corpora are not usable for neural machine translation training (i.e., the resulting…
Machine TranslationTranslationTransliterationWord AlignmentNRC Parallel Corpus Filtering System for WMT 2019
We describe the National Research Council Canada team{'}s submissions to the parallel corpus filtering task at the Fourth Conference on Machine Translation.
Machine TranslationTranslation