With Malice Towards None: Assessing Uncertainty via Equalized Coverage
An important factor to guarantee a fair use of data-driven recommendation systems is that we should be able to communicate their uncertainty to decision makers. This can be accomplished by constructing prediction intervals, which provide an intuitive measure of the limits of predictive performance. To support equitable treatment, we force the construction of such intervals to be unbiased in the sense that their coverage must be equal across all protected groups of interest. We present an operational methodology that achieves this goal by offering rigorous distribution-free coverage guarantees holding in finite samples. Our methodology, equalized coverage, is flexible as it can be viewed as a wrapper around any predictive algorithm. We test the applicability of the proposed framework on real data, demonstrating that equalized coverage constructs unbiased prediction intervals, unlike competitive methods.
Code (1)
Tasks
Prediction IntervalsRecommendation SystemsSimilar Papers 제목 키워드 기반
Conformal Classification with Equalized Coverage for Adaptively Selected Groups
This paper introduces a conformal inference method to evaluate uncertainty in classification by generating prediction sets with valid coverage conditional on adaptively chosen features. These features are carefully selec…
FairnessvalidConformal Prediction Sets Can Cause Disparate Impact
Although conformal prediction is a promising method for quantifying the uncertainty of machine learning models, the prediction sets it outputs are not inherently actionable. Many applications require a single output to a…
Conformal PredictionFairnessPredictionEqual Opportunity of Coverage in Fair Regression
We study fair machine learning (ML) under predictive uncertainty to enable reliable and trustworthy decision-making. The seminal work of 'equalized coverage' proposed an uncertainty-aware fairness notion. However, it doe…
Conformalizing Machine Translation Evaluation
Several uncertainty estimation methods have been recently proposed for machine translation evaluation. While these methods can provide a useful indication of when not to trust model predictions, we show in this paper tha…
Conformal PredictionMachine TranslationPredictionTranslationUnravelling the (In)compatibility of Statistical-Parity and Equalized-Odds
A key challenge in employing data, algorithms and data-driven systems is to adhere to the principle of fairness and justice. Statistical fairness measures belong to an important category of technical/formal mechanisms fo…