Validation of an Automatic Metric for the Accuracy of Pronoun Translation (APT)
In this paper, we define and assess a reference-based metric to evaluate the accuracy of pronoun translation (APT). The metric automatically aligns a candidate and a reference translation using GIZA++ augmented with specific heuristics, and then counts the number of identical or different pronouns, with provision for legitimate variations and omitted pronouns. All counts are then combined into one score. The metric is applied to the results of seven systems (including the baseline) that participated in the DiscoMT 2015 shared task on pronoun translation from English to French. The APT metric reaches around 0.993-0.999 Pearson correlation with human judges (depending on the parameters of APT), while other automatic metrics such as BLEU, METEOR, or those specific to pronouns used at DiscoMT 2015 reach only 0.972-0.986 Pearson correlation.
Code (1)
Tasks
Machine TranslationTranslationSimilar Papers 제목 키워드 기반
Automatic Reference-Based Evaluation of Pronoun Translation Misses the Point
We compare the performance of the APT and AutoPRF metrics for pronoun translation against a manually annotated dataset comprising human judgements as to the correctness of translations of the PROTEST test suite. Although…
TranslationA Large-Scale Test Set for the Evaluation of Context-Aware Pronoun Translation in Neural Machine Translation
The translation of pronouns presents a special challenge to machine translation to this day, since it often requires context outside the current sentence. Recent work on models that have access to information across sent…
Machine TranslationSentenceTranslationEvaluation Dataset for Zero Pronoun in Japanese to English Translation
In natural language, we often omit some words that are easily understandable from the context. In particular, pronouns of subject, object, and possessive cases are often omitted in Japanese; these are known as zero prono…
Machine TranslationTranslationQuality Estimation based Feedback Training for Improving Pronoun Translation
Pronoun translation is a longstanding challenge in neural machine translation (NMT), often requiring inter-sentential context to ensure linguistic accuracy. To address this, we introduce ProNMT, a novel framework designe…
Machine TranslationNMTTranslationDocument-level Neural MT: A Systematic Comparison
In this paper we provide a systematic comparison of existing and new document-level neural machine translation solutions. As part of this comparison, we introduce and evaluate a document-level variant of the recently pro…
Machine TranslationTranslation