Evaluating Gender Bias in German Machine Translation
We present WinoMTDE, a new gender bias evaluation test set designed to assess occupational stereotyping and underrepresentation in German machine translation (MT) systems. Building on the automatic evaluation method introduced by arXiv:1906.00591v1 [cs.CL], we extend the approach to German, a language with grammatical gender. The WinoMTDE dataset comprises 288 German sentences that are balanced in regard to gender, as well as stereotype, which was annotated using German labor statistics. We conduct a large-scale evaluation of five widely used MT systems and a large language model. Our results reveal persistent bias in most models, with the LLM outperforming traditional systems. The dataset and evaluation code are publicly available under https://github.com/michellekappl/mt_gender_german.
Code (1)
Tasks
Language ModelingLanguage ModellingLarge Language ModelMachine TranslationTranslationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
GG-BBQ: German Gender Bias Benchmark for Question Answering
Within the context of Natural Language Processing (NLP), fairness evaluation is often associated with the assessment of bias and reduction of associated harm. In this regard, the evaluation is usually carried out by usin…
Machine TranslationQuestion AnsweringBuilding Bridges: A Dataset for Evaluating Gender-Fair Machine Translation into German
The translation of gender-neutral person-referring terms (e.g., the students) is often non-trivial. Translating from English into German poses an interesting case -- in German, person-referring nouns are usually gender-s…
Machine TranslationSentenceTranslationDoes Context Help Mitigate Gender Bias in Neural Machine Translation?
Neural Machine Translation models tend to perpetuate gender bias present in their training data distribution. Context-aware models have been previously suggested as a means to mitigate this type of bias. In this work, we…
Machine TranslationTranslationMitigating Gender Bias in Machine Translation through Adversarial Learning
Machine translation and other NLP systems often contain significant biases regarding sensitive attributes, such as gender or race, that worsen system performance and perpetuate harmful stereotypes. Recent preliminary res…
Machine TranslationTranslationMitigating Gender Bias in Machine Translation through Adversarial Learning
Machine translation and other NLP systems often contain significant biases regarding sensitive attributes, such as gender or race, that worsen system performance and perpetuate harmful stereotypes. Recent preliminary res…
Machine TranslationTranslation