OpusFilter: A Configurable Parallel Corpus Filtering Toolbox

2020-07-01 · ACL 2020 6 · Mikko Aulamo, Sami Virpioja, J{\"o}rg Tiedemann

This paper introduces OpusFilter, a flexible and modular toolbox for filtering parallel corpora. It implements a number of components based on heuristic filters, language identification libraries, character-based language models, and word alignment tools, and it can easily be extended with custom filters. Bitext segments can be ranked according to their quality or domain match using single features or a logistic regression model that can be trained without manually labeled training data. We demonstrate the effectiveness of OpusFilter on the example of a Finnish-English news translation task based on noisy web-crawled training data. Applying our tool leads to improved translation quality while significantly reducing the size of the training data, also clearly outperforming an alternative ranking given in the crawled data set. Furthermore, we show the ability of OpusFilter to perform data selection for domain adaptation.

📄 PDF Abstract BibTeX

Code (0)

등록된 구현이 없습니다.

Tasks

Domain Adaptation Language Identification Translation Word Alignment

Methods 이 논문이 사용한 방법론

Logistic Regression Logistic Regression, despite its name, is a linear model for classification rather than regression. Logistic regression is also known in the literature as logit regression,…

ToFFi -- Toolbox for Frequency-based Fingerprinting of Brain Signals

2021-10-19 · Michał K. Komorowski, Krzysztof Rykaczewski, Tomasz Piotrowski, Katarzyna Jurewicz 외

Spectral fingerprints (SFs) are unique power spectra signatures of human brain regions of interest (ROIs, Keitel & Gross, 2016). SFs allow for accurate ROI identification and can serve as biomarkers of differences exhibi…

EEG Electroencephalogram (EEG)

Alibaba Submission to the WMT18 Parallel Corpus Filtering Task

2018-10-01 · WS 2018 10 · Jun Lu, Xiaoyu Lv, Yangbin Shi, Boxing Chen

This paper describes the Alibaba Machine Translation Group submissions to the WMT 2018 Shared Task on Parallel Corpus Filtering. While evaluating the quality of the parallel corpus, the three characteristics of the corpu…

Diversity Machine Translation Sentence Translation+1

Coverage and Cynicism: The AFRL Submission to the WMT 2018 Parallel Corpus Filtering Task

2018-10-01 · WS 2018 10 · Grant Erdmann, Jeremy Gwinnup

The WMT 2018 Parallel Corpus Filtering Task aims to test various methods of filtering a noisy parallel corpus, to make it useful for training machine translation systems. We describe the AFRL submissions, including their…

Machine Translation Translation

Accurate semantic textual similarity for cleaning noisy parallel corpora using semantic machine translation evaluation metric: The NRC supervised submissions to the Parallel Corpus Filtering task

2018-10-01 · WS 2018 10 · Chi-kiu Lo, Michel Simard, Darlene Stewart, Samuel Larkin 외

We present our semantic textual similarity approach in filtering a noisy web crawled parallel corpus using YiSi{---}a novel semantic machine translation evaluation metric. The systems mainly based on this supervised appr…

Machine Translation Semantic Textual Similarity Translation

NRC Parallel Corpus Filtering System for WMT 2019

2019-08-01 · WS 2019 8 · Gabriel Bernier-Colborne, Chi-kiu Lo

We describe the National Research Council Canada team{'}s submissions to the parallel corpus filtering task at the Fourth Conference on Machine Translation.