A Simple Way to Deal with Cherry-picking
Statistical hypothesis testing serves as statistical evidence for scientific innovation. However, if the reported results are intentionally biased, hypothesis testing no longer controls the rate of false discovery. In particular, we study such selection bias in machine learning models where the reporter is motivated to promote an algorithmic innovation. When the number of possible configurations (e.g., datasets) is large, we show that the reporter can falsely report an innovation even if there is no improvement at all. We propose a `post-reporting' solution to this issue where the bias of the reported results is verified by another set of results. The theoretical findings are supported by experimental results with synthetic and real-world datasets.
Code (0)
등록된 구현이 없습니다.
Tasks
Selection biasTwo-sample testingSimilar Papers 제목 키워드 기반
On the existence of a cherry-picking sequence
Recently, the minimum number of reticulation events that is required to simultaneously embed a collection P of rooted binary phylogenetic trees into a so-called temporal network has been characterized in terms of cherry-…
On Context-aware Detection of Cherry-picking in News Reporting
Cherry-picking refers to the deliberate selection of evidence or facts that favor a particular viewpoint while ignoring or distorting evidence that supports an opposing perspective. Manually identifying cherry-picked sta…
Cherry on the Cake: Fairness is NOT an Optimization Problem
In Fair AI literature, the practice of maliciously creating unfair models that nevertheless satisfy fairness constraints is known as "cherry-picking". A cherry-picking model is a model that makes mistakes on purpose, sel…
FairnessMulti-Label ClassificationMUlTI-LABEL-ClASSIFICATIONOn the Definition and Detection of Cherry-Picking in Counterfactual Explanations
Counterfactual explanations are widely used to communicate how inputs must change for a model to alter its prediction. For a single instance, many valid counterfactuals can exist, which leaves open the possibility for an…
Breaking the Trend: How to Avoid Cherry-Picked Signals
Our empirical results, illustrated in Fig.5, show an impressive fit with the pretty complex theoritical Sharpe formula of a Trend following strategy depending on the parameter of the signal, which was derived by Grebenko…