Measure Twice, Cut Once: Quantifying Bias and Fairness in Deep Neural Networks
Algorithmic bias is of increasing concern, both to the research community, and society at large. Bias in AI is more abstract and unintuitive than traditional forms of discrimination and can be more difficult to detect and mitigate. A clear gap exists in the current literature on evaluating the relative bias in the performance of multi-class classifiers. In this work, we propose two simple yet effective metrics, Combined Error Variance (CEV) and Symmetric Distance Error (SDE), to quantitatively evaluate the class-wise bias of two models in comparison to one another. By evaluating the performance of these new metrics and by demonstrating their practical application, we show that they can be used to measure fairness as well as bias. These demonstrations show that our metrics can address specific needs for measuring bias in multi-class classification.
Code (0)
등록된 구현이 없습니다.
Tasks
FairnessMulti-class ClassificationSimilar Papers 제목 키워드 기반
Standardized Interpretable Fairness Measures for Continuous Risk Scores
We propose a standardized version of fairness measures for continuous scores with a reasonable interpretation based on the Wasserstein distance. Our measures are easily computable and well suited for quantifying and inte…
FairnessUncovering Fairness through Data Complexity as an Early Indicator
Fairness constitutes a concern within machine learning (ML) applications. Currently, there is no study on how disparities in classification complexity between privileged and unprivileged groups could influence the fairne…
FairnessQuantifying Social Biases Using Templates is Unreliable
Recently, there has been an increase in efforts to understand how large language models (LLMs) propagate and amplify social biases. Several works have utilized templates for fairness evaluation, which allow researchers t…
AttributeBenchmarkingDiagnosticFairnessQuantifying Social Biases in NLP: A Generalization and Empirical Comparison of Extrinsic Fairness Metrics
Measuring bias is key for better understanding and addressing unfairness in NLP/ML models. This is often done via fairness metrics which quantify the differences in a model's behaviour across a range of demographic group…
FairnessIdentifying biases in legal data: An algorithmic fairness perspective
The need to address representation biases and sentencing disparities in legal case data has long been recognized. Here, we study the problem of identifying and measuring biases in large-scale legal case data from an algo…
Fairnessregression