The Power of Tests for Detecting $p$-Hacking
$p$-Hacking undermines the validity of empirical studies. A flourishing empirical literature investigates the prevalence of $p$-hacking based on the distribution of $p$-values across studies. Interpreting results in this literature requires a careful understanding of the power of methods for detecting $p$-hacking. We theoretically study the implications of likely forms of $p$-hacking on the distribution of $p$-values to understand the power of tests for detecting it. Power depends crucially on the $p$-hacking strategy and the distribution of true effects. Publication bias can enhance the power for testing the joint null of no $p$-hacking and no publication bias.
Code (1)
Similar Papers 제목 키워드 기반
Detecting p-hacking
We theoretically analyze the problem of testing for $p$-hacking based on distributions of $p$-values across multiple studies. We provide general results for when such distributions have testable restrictions (are non-inc…
EvilGenie: A Reward Hacking Benchmark
We introduce EvilGenie, a benchmark for reward hacking in programming settings. We source problems from LiveCodeBench and create an environment in which agents can easily reward hack, such as by hardcoding test cases or …
Reproducing, Analyzing, and Detecting Reward Hacking in Rubric-Based Reinforcement Learning
Rubric-based reinforcement learning (RL) uses an LLM-as-a-Judge (LaaJ) to score model outputs according to rubrics as rewards. However, policy models may exploit latent biases in the judge, leading to reward hacking and …
Reinforcement LearningDetecting and Suppressing Reward Hacking with Gradient Fingerprints
Reinforcement learning with verifiable rewards (RLVR) typically optimizes for outcome rewards without imposing constraints on intermediate reasoning. This leaves training susceptible to reward hacking, where models explo…
Reinforcement LearningLogical ReasoningSpecBench: Measuring Reward Hacking in Long-Horizon Coding Agents
As long-horizon coding agents produce more code than any developer can review, oversight collapses onto a single surface: the automated test suite. Reward hacking naturally arises in this setup, as the agent optimizes fo…