Investigating Human + Machine Complementarity for Recidivism Predictions
When might human input help (or not) when assessing risk in fairness domains? Dressel and Farid (2018) asked Mechanical Turk workers to evaluate a subset of defendants in the ProPublica COMPAS data for risk of recidivism, and concluded that COMPAS predictions were no more accurate or fair than predictions made by humans. We delve deeper into this claim to explore differences in human and algorithmic decision making. We construct a Human Risk Score based on the predictions made by multiple Turk workers, characterize the features that determine agreement and disagreement between COMPAS and Human Scores, and construct hybrid Human+Machine models to predict recidivism. Our key finding is that on this data set, Human and COMPAS decision making differed, but not in ways that could be leveraged to significantly improve ground-truth prediction. We present the results of our analyses and suggestions for data collection best practices to leverage complementary strengths of human and machines in the fairness domain.
Code (0)
등록된 구현이 없습니다.
Tasks
Decision MakingFairnessSimilar Papers 제목 키워드 기반
Harnessing Explanations to Bridge AI and Humans
Machine learning models are increasingly integrated into societally critical applications such as recidivism prediction and medical diagnosis, thanks to their superior predictive power. In these applications, however, fu…
Decision MakingMedical DiagnosisHow Aligned are Generative Models to Humans in High-Stakes Decision-Making?
Large generative models (LMs) are increasingly being considered for high-stakes decision-making. This work considers how such models compare to humans and predictive AI models on a specific case of recidivism prediction.…
Decision MakingIn-Context LearningOn Human Predictions with Explanations and Predictions of Machine Learning Models: A Case Study on Deception Detection
Humans are the final decision makers in critical tasks that involve ethical and legal concerns, ranging from recidivism prediction, to medical diagnosis, to fighting against fake news. Although machine learning models ca…
BIG-bench Machine LearningDeception DetectionMedical DiagnosisIn Pursuit of Interpretable, Fair and Accurate Machine Learning for Criminal Recidivism Prediction
Objectives: We study interpretable recidivism prediction using machine learning (ML) models and analyze performance in terms of prediction ability, sparsity, and fairness. Unlike previous works, this study trains interpr…
BIG-bench Machine LearningFairnessInterpretable Machine LearningRobust Human-AI Complementarity under Uncertainty
Machine learning models are often intended to augment rather than replace human decision makers, by providing information that is complementary to human judgement. Yet, in practice, human decision makers routinely fail t…