Bias Beyond English: Counterfactual Tests for Bias in Sentiment Analysis in Four Languages
Sentiment analysis (SA) systems are used in many products and hundreds of languages. Gender and racial biases are well-studied in English SA systems, but understudied in other languages, with few resources for such studies. To remedy this, we build a counterfactual evaluation corpus for gender and racial/migrant bias in four languages. We demonstrate its usefulness by answering a simple but important question that an engineer might need to answer when deploying a system: What biases do systems import from pre-trained models when compared to a baseline with no pre-training? Our evaluation corpus, by virtue of being counterfactual, not only reveals which models have less bias, but also pinpoints changes in model bias behaviour, which enables more targeted mitigation strategies. We release our code and evaluation corpora to facilitate future research.
Code (0)
등록된 구현이 없습니다.
Tasks
counterfactualSentiment AnalysisSimilar Papers 제목 키워드 기반
Double Perturbation: On the Robustness of Robustness and Counterfactual Bias Evaluation
Robustness and counterfactual bias are usually evaluated on a test dataset. However, are these evaluations robust? If the test dataset is perturbed slightly, will the evaluation results keep the same? In this paper, we p…
counterfactualPredictionCross-lingual Transfer Can Worsen Bias in Sentiment Analysis
Sentiment analysis (SA) systems are widely deployed in many of the world's languages, and there is well-documented evidence of demographic bias in these systems. In languages beyond English, scarcer training data is ofte…
counterfactualCross-Lingual TransferSentiment AnalysisTransfer LearningBias in Language Models: Beyond Trick Tests and Toward RUTEd Evaluation
Standard benchmarks of bias and fairness in large language models (LLMs) measure the association between the user attributes stated or implied by a prompt and the LLM's short text response, but human-AI interaction incre…
FairnessText GenerationBPL: Bias-adaptive Preference Distillation Learning for Recommender System
Recommender systems suffer from biases that cause the collected feedback to incompletely reveal user preference. While debiasing learning has been extensively studied, they mostly focused on the specialized (called count…
Reasoning Beyond Bias: A Study on Counterfactual Prompting and Chain of Thought Reasoning
Language models are known to absorb biases from their training data, leading to predictions driven by statistical regularities rather than semantic relevance. We investigate the impact of these biases on answer choice pr…
counterfactualMMLUMulti-task Language Understanding