Regional Negative Bias in Word Embeddings Predicts Racial Animus--but only via Name Frequency
The word embedding association test (WEAT) is an important method for measuring linguistic biases against social groups such as ethnic minorities in large text corpora. It does so by comparing the semantic relatedness of words prototypical of the groups (e.g., names unique to those groups) and attribute words (e.g., 'pleasant' and 'unpleasant' words). We show that anti-black WEAT estimates from geo-tagged social media data at the level of metropolitan statistical areas strongly correlate with several measures of racial animus--even when controlling for sociodemographic covariates. However, we also show that every one of these correlations is explained by a third variable: the frequency of Black names in the underlying corpora relative to White names. This occurs because word embeddings tend to group positive (negative) words and frequent (rare) words together in the estimated semantic space. As the frequency of Black names on social media is strongly correlated with Black Americans' prevalence in the population, this results in spurious anti-Black WEAT estimates wherever few Black Americans live. This suggests that research using the WEAT to measure bias should consider term frequency, and also demonstrates the potential consequences of using black-box models like word embeddings to study human cognition and behavior.
Code (0)
등록된 구현이 없습니다.
Tasks
AttributeWord EmbeddingsSimilar Papers 제목 키워드 기반
A Transparent Framework for Evaluating Unintended Demographic Bias in Word Embeddings
Word embedding models have gained a lot of traction in the Natural Language Processing community, however, they suffer from unintended demographic biases. Most approaches to evaluate these biases rely on vector space bas…
FairnessWord EmbeddingsDouble-Hard Debias: Tailoring Word Embeddings for Gender Bias Mitigation
Word embeddings derived from human-generated corpora inherit strong gender bias which can be further amplified by downstream models. Some commonly adopted debiasing approaches, including the seminal Hard Debias algorithm…
Word Embeddings"Thy algorithm shalt not bear false witness": An Evaluation of Multiclass Debiasing Methods on Word Embeddings
With the vast development and employment of artificial intelligence applications, research into the fairness of these algorithms has been increased. Specifically, in the natural language processing domain, it has been sh…
FairnessWord EmbeddingsThe Undesirable Dependence on Frequency of Gender Bias Metrics Based on Word Embeddings
Numerous works use word embedding-based metrics to quantify societal biases and stereotypes in texts. Recent studies have found that word embeddings can capture semantic similarity but may be affected by word frequency. …
Semantic SimilaritySemantic Textual SimilarityWord EmbeddingsImproving Contrastive Learning of Sentence Embeddings with Case-Augmented Positives and Retrieved Negatives
Following SimCSE, contrastive learning based methods have achieved the state-of-the-art (SOTA) performance in learning sentence embeddings. However, the unsupervised contrastive learning methods still lag far behind the …
AttributeContrastive LearningLanguage ModelingLanguage Modelling+5