Gender Coreference and Bias Evaluation at WMT 2020
Gender bias in machine translation can manifest when choosing gender inflections based on spurious gender correlations. For example, always translating doctors as men and nurses as women. This can be particularly harmful as models become more popular and deployed within commercial systems. Our work presents the largest evidence for the phenomenon in more than 19 systems submitted to the WMT over four diverse target languages: Czech, German, Polish, and Russian. To achieve this, we use WinoMT, a recent automatic test suite which examines gender coreference and bias when translating from English to languages with grammatical gender. We extend WinoMT to handle two new languages tested in WMT: Polish and Czech. We find that all systems consistently use spurious correlations in the data rather than meaningful contextual information.
Code (2)
Tasks
Machine TranslationTranslationSimilar Papers 제목 키워드 기반
Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods
We introduce a new benchmark, WinoBias, for coreference resolution focused on gender bias. Our corpus contains Winograd-schema style sentences with entities corresponding to people referred by their occupation (e.g. the …
coreference-resolutionCoreference ResolutionData AugmentationCollecting a Large-Scale Gender Bias Dataset for Coreference Resolution and Machine Translation
Recent works have found evidence of gender bias in models of machine translation and coreference resolution using mostly synthetic diagnostic datasets. While these quantify bias in a controlled experiment, they often do …
coreference-resolutionCoreference ResolutionDiagnosticMachine Translation+1Gender Bias in Coreference Resolution
We present an empirical study of gender bias in coreference resolution systems. We first introduce a novel, Winograd schema-style set of minimal pair sentences that differ only by pronoun gender. With these "Winogender s…
coreference-resolutionCoreference ResolutionTransforming Dutch: Debiasing Dutch Coreference Resolution Systems for Non-binary Pronouns
Gender-neutral pronouns are increasingly being introduced across Western languages. Recent evaluations have however demonstrated that English NLP systems are unable to correctly process gender-neutral pronouns, with the …
coreference-resolutionCoreference ResolutioncounterfactualData AugmentationWinoPron: Revisiting English Winogender Schemas for Consistency, Coverage, and Grammatical Case
While measuring bias and robustness in coreference resolution are important goals, such measurements are only as good as the tools we use to measure them. Winogender Schemas (Rudinger et al., 2018) are an influential dat…
coreference-resolutionCoreference ResolutionLanguage Modelling