Gender-tuning: Empowering Fine-tuning for Debiasing Pre-trained Language Models
Recent studies have revealed that the widely-used Pre-trained Language Models (PLMs) propagate societal biases from the large unmoderated pre-training corpora. Existing solutions require debiasing training processes and datasets for debiasing, which are resource-intensive and costly. Furthermore, these methods hurt the PLMs' performance on downstream tasks. In this study, we propose Gender-tuning, which debiases the PLMs through fine-tuning on downstream tasks' datasets. For this aim, Gender-tuning integrates Masked Language Modeling (MLM) training objectives into fine-tuning's training process. Comprehensive experiments show that Gender-tuning outperforms the state-of-the-art baselines in terms of average gender bias scores in PLMs while improving PLMs' performance on downstream tasks solely using the downstream tasks' dataset. Also, Gender-tuning is a deployable debiasing tool for any PLM that works with original fine-tuning.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage ModellingMasked Language ModelingSimilar Papers 제목 키워드 기반
Unlabeled Debiasing in Downstream Tasks via Class-wise Low Variance Regularization
Language models frequently inherit societal biases from their training data. Numerous techniques have been proposed to mitigate these biases during both the pre-training and fine-tuning stages. However, fine-tuning a pre…
AttributeLanguage ModelingLanguage ModellingDoes Debiasing Inevitably Degrade the Model Performance
Gender bias in language models has attracted sufficient attention because it threatens social justice. However, most of the current debiasing methods degraded the model's performance on other tasks while the degradation …
modelDebiasing Gender Bias in Information Retrieval Models
Biases in culture, gender, ethnicity, etc. have existed for decades and have affected many areas of human social interaction. These biases have been shown to impact machine learning (ML) models, and for natural language …
ArticlesCultural Vocal Bursts Intensity PredictionInformation RetrievalRetrievalBendVLM: Test-Time Debiasing of Vision-Language Embeddings
Vision-language model (VLM) embeddings have been shown to encode biases present in their training data, such as societal biases that prescribe negative characteristics to members of various racial and gender identities. …
AttributeImage GenerationLanguage ModelingLanguage ModellingRevealing and Reducing Gender Biases in Vision and Language Assistants (VLAs)
Pre-trained large language models (LLMs) have been reliably integrated with visual input for multimodal tasks. The widespread adoption of instruction-tuned image-to-text vision-language assistants (VLAs) like LLaVA and I…
AttributeImage to text