HONEST: Measuring Hurtful Sentence Completion in Language Models
Language models have revolutionized the field of NLP. However, language models capture and proliferate hurtful stereotypes, especially in text generation. Our results show that 4.3{\%} of the time, language models complete a sentence with a hurtful word. These cases are not random, but follow language and gender-specific patterns. We propose a score to measure hurtful sentence completions in language models (HONEST). It uses a systematic template- and lexicon-based bias evaluation methodology for six languages. Our findings suggest that these models replicate and amplify deep-seated societal stereotypes about gender roles. Sentence completions refer to sexual promiscuity when the target is female in 9{\%} of the time, and in 4{\%} to homosexuality when the target is male. The results raise questions about the use of these models in production settings.
Code (1)
Tasks
Hate Speech DetectionHurtful Sentence CompletionMultilingual NLPSentenceSentence CompletionText GenerationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Measuring Gender Bias in West Slavic Language Models
Pre-trained language models have been known to perpetuate biases from the underlying datasets to downstream tasks. However, these findings are predominantly based on monolingual language models for English, whereas there…
Language ModellingMeasuring Harmful Sentence Completion in Language Models for LGBTQIA+ Individuals
Current language technology is ubiquitous and directly influences individuals’ lives worldwide. Given the recent trend in AI on training and constantly releasing new and powerful large language models (LLMs), there is a …
SentenceSentence CompletionMeasuring Harmful Representations in Scandinavian Language Models
Scandinavian countries are perceived as role-models when it comes to gender equality. With the advent of pre-trained language models and their widespread usage, we investigate to what extent gender-based harmful and toxi…
FairBelief - Assessing Harmful Beliefs in Language Models
Language Models (LMs) have been shown to inherit undesired biases that might hurt minorities and underrepresented groups if such systems were integrated into real-world applications without careful fairness auditing. Thi…
FairnessShowing LLM-Generated Code Selectively Based on Confidence of LLMs
Large Language Models (LLMs) have shown impressive abilities in code generation, but they may generate erroneous programs. Reading a program takes ten times longer than writing it. Showing these erroneous programs to dev…
Code Generation