A Predictive Factor Analysis of Social Biases and Task-Performance in Pretrained Masked Language Models
Various types of social biases have been reported with pretrained Masked Language Models (MLMs) in prior work. However, multiple underlying factors are associated with an MLM such as its model size, size of the training data, training objectives, the domain from which pretraining data is sampled, tokenization, and languages present in the pretrained corpora, to name a few. It remains unclear as to which of those factors influence social biases that are learned by MLMs. To study the relationship between model factors and the social biases learned by an MLM, as well as the downstream task performance of the model, we conduct a comprehensive study over 39 pretrained MLMs covering different model sizes, training objectives, tokenization methods, training data domains and languages. Our results shed light on important factors often neglected in prior literature, such as tokenization or model objectives.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
A Multidimensional Analysis of Social Biases in Vision Transformers
The embedding spaces of image models have been shown to encode a range of social biases such as racism and sexism. Here, we investigate specific factors that contribute to the emergence of these biases in Vision Transfor…
counterfactualFairnessUnveiling and Mitigating Bias in Mental Health Analysis with Large Language Models
The advancement of large language models (LLMs) has demonstrated strong capabilities across various applications, including mental health analysis. However, existing studies have focused on predictive performance, leavin…
FairnessQualitative Analysis of Depression Models by Demographics
Models for identifying depression using social media text exhibit biases towards different gender and racial/ethnic groups. Factors like representation and balance of groups within the dataset are contributory factors, b…
The Lifecycle of "Facts": A Survey of Social Bias in Knowledge Graphs
Knowledge graphs are increasingly used in a plethora of downstream tasks or in the augmentation of statistical models to improve factuality. However, social biases are engraved in these representations and propagate down…
Knowledge GraphsRelating Word Embedding Gender Biases to Gender Gaps: A Cross-Cultural Analysis
Modern models for common NLP tasks often employ machine learning techniques and train on journalistic, social media, or other culturally-derived text. These have recently been scrutinized for racial and gender biases, ro…
Cultural Vocal Bursts Intensity PredictionWord Embeddings