Investigating Failures of Automatic Translation in the Case of Unambiguous Gender
Transformer based models are the modern work horses for neural machine translation (NMT), reaching state of the art across several benchmarks. Despite their impressive accuracy, we observe a systemic and rudimentary class of errors made by transformer based models with regards to translating from a language that doesn't mark gender on nouns into others that do. We find that even when the surrounding context provides unambiguous evidence of the appropriate grammatical gender marking, no transformer based model we tested was able to accurately gender occupation nouns systematically. We release an evaluation scheme and dataset for measuring the ability of transformer based NMT models to translate gender morphology correctly in unambiguous contexts across syntactically diverse sentences. Our dataset translates from an English source into 20 languages from several different language families. With the availability of this dataset, our hope is that the NMT community can iterate on solutions for this class of especially egregious errors.
Code (0)
등록된 구현이 없습니다.
Tasks
Machine TranslationNMTTranslationSimilar Papers 제목 키워드 기반
Gender, names and other mysteries: Towards the ambiguous for gender-inclusive translation
The vast majority of work on gender in MT focuses on 'unambiguous' inputs, where gender markers in the source language are expected to be resolved in the output. Conversely, this paper explores the widespread case where …
SentenceTranslationInvestigating Length Issues in Document-level Machine Translation
Transformer architectures are increasingly effective at processing and generating very long chunks of texts, opening new perspectives for document-level machine translation (MT). In this work, we challenge the ability of…
Document Level Machine TranslationMachine TranslationSentenceTranslationGender Bias in English-to-Greek Machine Translation
As the demand for inclusive language increases, concern has grown over the susceptibility of machine translation (MT) systems to reinforce gender stereotypes. This study investigates gender bias in two commercial MT syst…
Machine TranslationTranslationTesting Untestable Neural Machine Translation: An Industrial Case
Neural Machine Translation (NMT) has been widely adopted recently due to its advantages compared with the traditional Statistical Machine Translation (SMT). However, an NMT system still often produces translation failure…
Machine TranslationNMTTranslationvalidUncertainty Quantification for Evaluating Machine Translation Bias
The predictive uncertainty of machine translation (MT) models is typically used as a quality estimation proxy. In this work, we posit that apart from confidently translating when a single correct translation exists, mode…
Machine Translation