Selecting the best data filtering method for NMT training
Performance of NMT systems has been proven to depend on the quality of the training data. In this paper we explore different open-source tools that can be used to score the quality of translation pairs, with the goal of obtaining clean corpora for training NMT models. We measure the performance of these tools by correlating their scores with human scores, as well as rank models trained on the resulting filtered datasets in terms of their performance on different test sets and MT performance metrics.
Code (0)
등록된 구현이 없습니다.
Tasks
NMTTranslationSimilar Papers 제목 키워드 기반
Joint Out-of-Distribution Filtering and Data Discovery Active Learning
As the data demand for deep learning models increases, active learning (AL) becomes essential to strategically select samples for labeling, which maximizes data efficiency and reduces training costs. Real-world scena…
Active LearningTradeoffs in Resampling and Filtering for Imbalanced Classification
Imbalanced classification problems are extremely common in natural language processing and are solved using a variety of resampling and filtering techniques, which often involve making decisions on how to select training…
Classificationimbalanced classificationRetrievaltoken-classification+1Active Learning for Entity Filtering in Microblog Streams
Monitoring the reputation of entities such as companies or brands in microblog streams (e.g., Twitter) starts by selecting mentions that are related to the entity of interest. Entities are often ambiguous (e.g., "Jaguar'…
Active LearningData Filtering using Cross-Lingual Word Embeddings
Data filtering for machine translation (MT) describes the task of selecting a subset of a given, possibly noisy corpus with the aim to maximize the performance of an MT system trained on this selected data. Over the year…
Cross-Lingual Word EmbeddingsLanguage IdentificationMachine TranslationWord EmbeddingsConfidence through Attention
Attention distributions of the generated translations are a useful bi-product of attention-based recurrent neural network translation models and can be treated as soft alignments between the input and output tokens. In t…
Machine TranslationTranslation