Measuring Gender Bias in Job Title Matching for Grammatical Gender Languages
This work sets the ground for studying how explicit grammatical gender assignment in job titles can affect the results of automatic job ranking systems. We propose the usage of metrics for ranking comparison controlling for gender to evaluate gender bias in job title ranking systems, in particular RBO (Rank-Biased Overlap). We generate and share test sets for a job title matching task in four grammatical gender languages, including occupations in masculine and feminine form and annotated by gender and matching relevance. We use the new test sets and the proposed methodology to evaluate the gender bias of several out-of-the-box multilingual models to set as baselines, showing that all of them exhibit varying degrees of gender bias.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Measuring Gender Bias in Word Embeddings of Gendered Languages Requires Disentangling Grammatical Gender Signals
Does the grammatical gender of a language interfere when measuring the semantic gender information captured by its word embeddings? A number of anomalous gender bias measurements in the embeddings of gendered languages s…
Word EmbeddingsProText: A benchmark dataset for measuring (mis)gendering in long-form texts
We introduce ProText, a dataset for measuring gendering and misgendering in stylistically diverse long-form English texts. ProText spans three dimensions: Theme nouns (names, occupations, titles, kinship terms), Theme ca…
Evaluating Gender Bias Transfer from Film Data
Films are a rich source of data for natural language processing. OpenSubtitles (Lison and Tiedemann, 2016) is a popular movie script dataset, used for training models for tasks such as machine translation and dialogue ge…
Dialogue GenerationMachine TranslationSentenceSentence Embedding+2Measuring and Mitigating Name Biases in Neural Machine Translation
Neural Machine Translation (NMT) systems exhibit problematic biases, such as stereotypical gender bias in the translation of occupation terms into languages with grammatical gender. In this paper we describe a new source…
Data AugmentationMachine TranslationNMTTranslationEvaluating Gender Bias in Hindi-English Machine Translation
With language models being deployed increasingly in the real world, it is essential to address the issue of the fairness of their outputs. The word embedding representations of these language models often implicitly draw…
FairnessMachine TranslationSentenceTranslation