Test Suites Task: Evaluation of Gender Fairness in MT with MuST-SHE and INES
As part of the WMT-2023 "Test suites" shared task, in this paper we summarize the results of two test suites evaluations: MuST-SHE-WMT23 and INES. By focusing on the en-de and de-en language pairs, we rely on these newly created test suites to investigate systems' ability to translate feminine and masculine gender and produce gender-inclusive translations. Furthermore we discuss metrics associated with our test suites and validate them by means of human evaluations. Our results indicate that systems achieve reasonable and comparable performance in correctly translating both feminine and masculine gender forms for naturalistic gender phenomena. Instead, the generation of inclusive language forms in translation emerges as a challenging task for all the evaluated MT models, indicating room for future improvements and research on the topic.
Code (1)
Tasks
de-enFairnessSimilar Papers 제목 키워드 기반
FAIRE: Assessing Racial and Gender Bias in AI-Driven Resume Evaluations
In an era where AI-driven hiring is transforming recruitment practices, concerns about fairness and bias have become increasingly important. To explore these issues, we introduce a benchmark, FAIRE (Fairness Assessment I…
FairnessUnderneath the Numbers: Quantitative and Qualitative Gender Fairness in LLMs for Depression Prediction
Recent studies show bias in many machine learning models for depression detection, but bias in LLMs for this task remains unexplored. This work presents the first attempt to investigate the degree of gender bias present …
Depression DetectionFairnessAutomated Evaluation of Gender Bias Across 13 Large Multimodal Models
Large multimodal models (LMMs) have revolutionized text-to-image generation, but they risk perpetuating the harmful social biases in their training data. Prior work has identified gender bias in these models, but methodo…
Text-to-Image GenerationFairQE: Multi-Agent Framework for Mitigating Gender Bias in Translation Quality Estimation
Quality Estimation (QE) aims to assess machine translation quality without reference translations, but recent studies have shown that existing QE models exhibit systematic gender bias. In particular, they tend to favor m…
Machine TranslationPRIDE -- Parameter-Efficient Reduction of Identity Discrimination for Equality in LLMs
Large Language Models (LLMs) frequently reproduce the gender- and sexual-identity prejudices embedded in their training corpora, leading to outputs that marginalize LGBTQIA+ users. Hence, reducing such biases is of great…
parameter-efficient fine-tuning