Integrating Expert Judgment and Algorithmic Decision Making: An Indistinguishability Framework
We introduce a novel framework for human-AI collaboration in prediction and decision tasks. Our approach leverages human judgment to distinguish inputs which are algorithmically indistinguishable, or "look the same" to any feasible predictive algorithm. We argue that this framing clarifies the problem of human-AI collaboration in prediction and decision tasks, as experts often form judgments by drawing on information which is not encoded in an algorithm's training data. Algorithmic indistinguishability yields a natural test for assessing whether experts incorporate this kind of "side information", and further provides a simple but principled method for selectively incorporating human feedback into algorithmic predictions. We show that this method provably improves the performance of any feasible algorithmic predictor and precisely quantify this improvement. We demonstrate the utility of our framework in a case study of emergency room triage decisions, where we find that although algorithmic risk scores are highly competitive with physicians, there is strong evidence that physician judgments provide signal which could not be replicated by any predictive algorithm. This insight yields a range of natural decision rules which leverage the complementary strengths of human experts and predictive algorithms.
Code (1)
Tasks
Decision MakingSimilar Papers 제목 키워드 기반
Reputation-Driven Adoption and Avoidance of Algorithmic Decision Aids in Credence Goods Markets
In credence goods markets such as health care or repair services, consumers rely on experts with superior information to adequately diagnose and treat them. Experts, however, are constrained in their diagnostic abilities…
DiagnosticHuman Perceptions of Fairness in Algorithmic Decision Making: A Case Study of Criminal Risk Prediction
As algorithms are increasingly used to make important decisions that affect human lives, ranging from social benefit assignment to predicting risk of criminal recidivism, concerns have been raised about the fairness of a…
Decision MakingFairnessTowards secure judgments aggregation in AHP
In decision-making methods, it is common to assume that the experts are honest and professional. However, this is not the case when one or more experts in the group decision making framework, such as the group analytic h…
Decision MakingHuman Expertise in Algorithmic Prediction
We introduce a novel framework for incorporating human expertise into algorithmic predictions. Our approach leverages human judgment to distinguish inputs which are algorithmically indistinguishable, or "look the same" t…
PredictionX-ray ClassificationEvaluation and selection of Medical Tourism sites: A rough AHP based MABAC approach
In this paper, a novel multiple criteria decision making (MCDM) methodology is presented for assessing and prioritizing medical tourism destinations in uncertain environment. A systematic evaluation and assessment method…
Decision Making