Flexible text generation for counterfactual fairness probing
A common approach for testing fairness issues in text-based classifiers is through the use of counterfactuals: does the classifier output change if a sensitive attribute in the input is changed? Existing counterfactual generation methods typically rely on wordlists or templates, producing simple counterfactuals that don't take into account grammar, context, or subtle sensitive attribute references, and could miss issues that the wordlist creators had not considered. In this paper, we introduce a task for generating counterfactuals that overcomes these shortcomings, and demonstrate how large language models (LLMs) can be leveraged to make progress on this task. We show that this LLM-based method can produce complex counterfactuals that existing methods cannot, comparing the performance of various counterfactual generation methods on the Civil Comments dataset and showing their value in evaluating a toxicity classifier.
Code (0)
등록된 구현이 없습니다.
Tasks
AttributecounterfactualFairnessText GenerationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Counterfactual Multi-Token Fairness in Text Classification
The counterfactual token generation has been limited to perturbing only a single token in texts that are generally short and single sentences. These tokens are often associated with one of many sensitive attributes. With…
AttributeClassificationcounterfactualFairness+3COFFEE: Counterfactual Fairness for Personalized Text Generation in Explainable Recommendation
As language models become increasingly integrated into our digital lives, Personalized Text Generation (PTG) has emerged as a pivotal component with a wide range of applications. However, the bias inherent in user writte…
counterfactualCounterfactual InferenceExplainable RecommendationExplanation Generation+2Analyzing Fairness of Neural Network Prediction via Counterfactual Dataset Generation
Interpreting the inference-time behavior of deep neural networks remains a challenging problem. Existing approaches to counterfactual explanation typically ask: What is the closest alternative input that would alter the …
GenderBias-\emph{VL}: Benchmarking Gender Bias in Vision Language Models via Counterfactual Probing
Large Vision-Language Models (LVLMs) have been widely adopted in various applications; however, they exhibit significant gender biases. Existing benchmarks primarily evaluate gender bias at the demographic group level, n…
BenchmarkingcounterfactualFairnessSemantic Similarity+1Counterfactual Fairness in Text Classification through Robustness
In this paper, we study counterfactual fairness in text classification, which asks the question: How would the prediction change if the sensitive attribute referenced in the example were different? Toxicity classifiers d…
AttributeClassificationcounterfactualFairness+3