[Re] Fairness Guarantees under Demographic Shift
Scope of Reproducibility: The original authors' main contribution is the family of Shifty algorithms, which can guarantee that certain fairness constraints will hold with high confidence even after a demographic shift in the deployment population occurs. They claim that Shifty provides these high-confidence fairness guarantees without a loss in model performance, given enough training data. Methodology: The code provided by the original paper was used, and only some small adjustments needed to be made in order to reproduce the experiments. All model specifications and hyperparameters from the original implementation were used. Extending beyond reproducing the original paper, we investigated the sensibility of Shifty to the size of the bounding intervals limiting the possible demographic shift, and ran shifty with an additional optimization method. Results: Our results approached the results reported in the original paper. They supported the claim that \textit{Shifty} reliably guarantees fairness under demographic shift, but could not verify that Shifty performs at no loss of accuracy. What was easy: The theoretical framework laid out in the original paper was well explained and supported by additional formulas and proofs in the appendix. Further, the authors provided clear instructions on how to run the experiments and provided necessary hyperparameters. What was difficult: While an open-source implementation of Shifty was provided and was debugged with relatively low time investment, the code did not contain extensive documentation and was complex to understand. It was therefore difficult to verify that each part of the code functions as expected and to expand upon the existing experiments. Further, certain hyperparameter and model specifications deviated between the provided code and the original paper, which made it challenging to know which specifications to apply when reproducing. Communication with original authors: The first author of the original paper was contacted, but we have yet to receive a reply.Paper Url: https://iclr.cc/virtual/2022/poster/6666Paper Review Url: https://openreview.net/forum?id=wbPObLm6ueAPaper Venue: ICLR 2022Confirmation: The report pdf is generated from the provided camera ready Google Colab script, The report metadata is verified from the camera ready Google Colab script, The report contains correct author information., The report contains link to code and SWH metadata., The report follows the ReScience latex style guides as in the Reproducibility Report Template (https://paperswithcode.com/rc2022/registration)., The report contains the Reproducibility Summary in the first page., The latex .zip file is verified from the camera ready Google Colab scriptLatex: zipJournal: ReScience Volume 9 Issue 2 Article 13Doi: https://www.doi.org/10.5281/zenodo.8173680Code: https://archive.softwareheritage.org/swh:1:dir:436c48ce9cf36b10ee3cdcd537a06c9df2cd53cc
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Fairness Guarantees under Demographic Shift
Recent studies have demonstrated that using machine learning for social applications can lead to injustice in the form of racist, sexist, and otherwise unfair and discriminatory outcomes. To address this challenge, recen…
FairnessFairness Under Group-Conditional Prior Probability Shift: Invariance, Drift, and Target-Aware Post-Processing
Machine learning systems are often trained and evaluated for fairness on historical data, yet deployed in environments where conditions have shifted. A particularly common form of shift occurs when the prevalence of posi…
Retiring $Δ$DP: New Distribution-Level Metrics for Demographic Parity
Demographic parity is the most widely recognized measure of group fairness in machine learning, which ensures equal treatment of different demographic groups. Numerous works aim to achieve demographic parity by pursuing …
FairnessStatistical Inference for Fairness Auditing
Before deploying a black-box model in high-stakes problems, it is important to evaluate the model's performance on sensitive subpopulations. For example, in a recidivism prediction task, we may wish to identify demograph…
FairnessProductionized Fairness Measurement Under Privacy Constraints
Fairness measurements in the form of disaggregated evaluations often rely on demographic signals that are legally constrained or culturally sensitive. Race and ethnicity signals are among the more difficult signals to cu…