ABROCA Distributions For Algorithmic Bias Assessment: Considerations Around Interpretation
Algorithmic bias continues to be a key concern of learning analytics. We study the statistical properties of the Absolute Between-ROC Area (ABROCA) metric. This fairness measure quantifies group-level differences in classifier performance through the absolute difference in ROC curves. ABROCA is particularly useful for detecting nuanced performance differences even when overall Area Under the ROC Curve (AUC) values are similar. We sample ABROCA under various conditions, including varying AUC differences and class distributions. We find that ABROCA distributions exhibit high skewness dependent on sample sizes, AUC differences, and class imbalance. When assessing whether a classifier is biased, this skewness inflates ABROCA values by chance, even when data is drawn (by simulation) from populations with equivalent ROC curves. These findings suggest that ABROCA requires careful interpretation given its distributional properties, especially when used to assess the degree of bias and when classes are imbalanced.
Code (1)
Tasks
FairnessSimilar Papers 제목 키워드 기반
Toward Sufficient Statistical Power in Algorithmic Bias Assessment: A Test for ABROCA
Algorithmic bias is a pressing concern in educational data mining (EDM), as it risks amplifying inequities in learning outcomes. The Area Between ROC Curves (ABROCA) metric is frequently used to measure discrepancies in …
FairnessIs Your Model "MADD"? A Novel Metric to Evaluate Algorithmic Fairness for Predictive Student Models
Predictive student models are increasingly used in learning environments due to their ability to enhance educational outcomes and support stakeholders in making informed decisions. However, predictive models can be biase…
FairnessWhither Bias Goes, I Will Go: An Integrative, Systematic Review of Algorithmic Bias Mitigation
Machine learning (ML) models are increasingly used for personnel assessment and selection (e.g., resume screeners, automatically scored interviews). However, concerns have been raised throughout society that ML assessmen…
FairnessMitigating Bias in Algorithmic Hiring: Evaluating Claims and Practices
There has been rapidly growing interest in the use of algorithms in hiring, especially as a means to address or mitigate bias. Yet, to date, little is known about how these methods are used in practice. How are algorithm…
Assessing Classifier Fairness with Collider Bias
The increasing application of machine learning techniques in everyday decision-making processes has brought concerns about the fairness of algorithmic decision-making. This paper concerns the problem of collider bias whi…
Decision MakingFairness