Real-time Program Evaluation using Anytime-valid Rank Tests
Counterfactual mean estimators such as difference-in-differences and synthetic control have grown into workhorse tools for program evaluation. Inference for these estimators is well-developed in settings where all post-treatment data is available at the time of analysis. However, in settings where data arrives sequentially, these tests do not permit real-time inference, as they require a pre-specified sample size T. We introduce real-time inference for program evaluation through anytime-valid rank tests. Our methodology relies on interpreting the absence of a treatment effect as exchangeability of the treatment estimates. We then convert these treatment estimates into sequential ranks, and construct optimal finite-sample valid sequential tests for exchangeability. We illustrate our methods in the context of difference-in-differences and synthetic control. In simulations, they control size even under mild exchangeability violations. While our methods suffer slight power loss at T, they allow for early rejection (before T) and preserve the ability to reject later (after T).
Code (0)
등록된 구현이 없습니다.
Tasks
counterfactualvalidSimilar Papers 제목 키워드 기반
CELEUS: Certifiable and Efficient LLM Evaluation via E-Processes
Can we trust evaluation scores to capture an LLM's true real-world performance? Certifiable evaluation answers this question by providing guarantee for LLM evaluation. In particular, existing methods sequentially curate …
Anytime Computation of Cautious Consequences in Answer Set Programming
Query answering in Answer Set Programming (ASP) is usually solved by computing (a subset of) the cautious consequences of a logic program. This task is computationally very hard, and there are programs for which computin…
PACE: Anytime-Valid Acceptance Tests for Self-Evolving Agents
Self-evolving agents improve by repeatedly proposing changes to their own prompts, skills, or workflows and keeping those that score higher on a small held-out set. Almost all effort has gone into the proposer that gener…
A Generic Complete Anytime Beam Search for Optimal Decision Tree
Finding an optimal decision tree that minimizes classification error is known to be NP-hard. While exact algorithms based on MILP, CP, SAT, or dynamic programming guarantee optimality, they often suffer from poor anytime…
AnyPose: Anytime 3D Human Pose Forecasting via Neural Ordinary Differential Equations
Anytime 3D human pose forecasting is crucial to synchronous real-world human-machine interaction, where the term ``anytime" corresponds to predicting human pose at any real-valued time step. However, to the best of our k…
Human Pose Forecasting