Measuring Social Biases of Crowd Workers using Counterfactual Queries
Social biases based on gender, race, etc. have been shown to pollute machine learning (ML) pipeline predominantly via biased training datasets. Crowdsourcing, a popular cost-effective measure to gather labeled training datasets, is not immune to the inherent social biases of crowd workers. To ensure such social biases aren't passed onto the curated datasets, it's important to know how biased each crowd worker is. In this work, we propose a new method based on counterfactual fairness to quantify the degree of inherent social bias in each crowd worker. This extra information can be leveraged together with individual worker responses to curate a less biased dataset.
Code (0)
등록된 구현이 없습니다.
Tasks
BIG-bench Machine LearningcounterfactualFairnessSimilar Papers 제목 키워드 기반
Demographic Biases of Crowd Workers in Key Opinion Leaders Finding
Key Opinion Leaders (KOLs) are people that have a strong influence and their opinions are listened to by people when making important decisions. Crowdsourcing provides an efficient and cost-effective means to gather data…
counterfactualCounterfactually Measuring and Eliminating Social Bias in Vision-Language Pre-training Models
Vision-Language Pre-training (VLP) models have achieved state-of-the-art performance in numerous cross-modal tasks. Since they are optimized to capture the statistical properties of intra- and inter-modality, there remai…
counterfactualScoring Workers in Crowdsourcing: How Many Control Questions are Enough?
We study the problem of estimating continuous quantities, such as prices, probabilities, and point spreads, using a crowdsourcing approach. A challenging aspect of combining the crowd's answers is that workers' reliabil…
Experimental Narratives: A Comparison of Human Crowdsourced Storytelling and AI Storytelling
The paper proposes a framework that combines behavioral and computational experiments employing fictional prompts as a novel tool for investigating cultural artifacts and social biases in storytelling both by humans and …
Mitigating Cognitive Biases in Multi-Criteria Crowd Assessment
Crowdsourcing is an easy, cheap, and fast way to perform large scale quality assessment; however, human judgments are often influenced by cognitive biases, which lowers their credibility. In this study, we focus on cogni…