Fair Enough: Standardizing Evaluation and Model Selection for Fairness Research in NLP
Modern NLP systems exhibit a range of biases, which a growing literature on model debiasing attempts to correct. However current progress is hampered by a plurality of definitions of bias, means of quantification, and oftentimes vague relation between debiasing algorithms and theoretical measures of bias. This paper seeks to clarify the current situation and plot a course for meaningful progress in fair learning, with two key contributions: (1) making clear inter-relations among the current gamut of methods, and their relation to fairness theory; and (2) addressing the practical problem of model selection, which involves a trade-off between fairness and accuracy and has led to systemic issues in fairness research. Putting them together, we make several recommendations to help shape future work.
Code (1)
Tasks
FairnessModel SelectionRelationSimilar Papers 제목 키워드 기반
A Human-in-the-Loop Fairness-Aware Model Selection Framework for Complex Fairness Objective Landscapes
Fairness-aware Machine Learning (FairML) applications are often characterized by complex social objectives and legal requirements, frequently involving multiple, potentially conflicting notions of fairness. Despite the w…
FairnessModel SelectionNavigateCertifying the Fairness of KNN in the Presence of Dataset Bias
We propose a method for certifying the fairness of the classification result of a widely used supervised learning algorithm, the k-nearest neighbors (KNN), under the assumption that the training data may have historical …
FairnessMEDFAIR: Benchmarking Fairness for Medical Imaging
A multitude of work has shown that machine learning-based medical diagnosis systems can be biased against certain subgroups of people. This has motivated a growing number of bias mitigation algorithms that aim to address…
BenchmarkingFairnessMedical DiagnosisModel SelectionFairGT: A Fairness-aware Graph Transformer
The design of Graph Transformers (GTs) generally neglects considerations for fairness, resulting in biased outcomes against certain sensitive subgroups. Since GTs encode graph information without relying on message-passi…
Fairnessfeature selectionGraph LearningUncertainty Quantification for Fairness in Two-Stage Recommender Systems
Many large-scale recommender systems consist of two stages. The first stage efficiently screens the complete pool of items for a small subset of promising candidates, from which the second-stage model curates the final r…
FairnessRecommendation SystemsUncertainty QuantificationVocal Bursts Valence Prediction