AExGym: Benchmarks and Environments for Adaptive Experimentation
Innovations across science and industry are evaluated using randomized trials (a.k.a. A/B tests). While simple and robust, such static designs are inefficient or infeasible for testing many hypotheses. Adaptive designs can greatly improve statistical power in theory, but they have seen limited adoption due to their fragility in practice. We present a benchmark for adaptive experimentation based on real-world datasets, highlighting prominent practical challenges to operationalizing adaptivity: non-stationarity, batched/delayed feedback, multiple outcomes and objectives, and external validity. Our benchmark aims to spur methodological development that puts practical performance (e.g., robustness) as a central concern, rather than mathematical guarantees on contrived instances. We release an open source library, AExGym, which is designed with modularity and extensibility in mind to allow experimentation practitioners to develop custom environments and algorithms.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
XLand-MiniGrid: Scalable Meta-Reinforcement Learning Environments in JAX
Inspired by the diversity and depth of XLand and the simplicity and minimalism of MiniGrid, we present XLand-MiniGrid, a suite of tools and grid-world environments for meta-reinforcement learning research. Written in JAX…
DiversityGPUMeta-LearningMeta Reinforcement Learning+2SPADE: Self-Play in Adaptive Synthetic Executable Environments
Continuous self-improvement requires an ever-expanding pool of self-generated, diverse, adaptive goals. For language agents, existing training environment pools (hand-curated, statically synthesized, or frozen-verifier) …
OpenAI GymJAXenstein: Accelerated Benchmarking for First-Person Environments
The progression of reinforcement learning algorithms have been driven by challenging benchmarks. The rate in which a researcher can iterate on a problem setting directly impacts the speed of algorithm development. Modern…
Reinforcement LearningAdaptive Experimentation with Delayed Binary Feedback
Conducting experiments with objectives that take significant delays to materialize (e.g. conversions, add-to-cart events, etc.) is challenging. Although the classical "split sample testing" is still valid for the delayed…
Multi-Armed BanditsvalidAdaptive Experimental Design and Counterfactual Inference
Adaptive experimental design methods are increasingly being used in industry as a tool to boost testing throughput or reduce experimentation cost relative to traditional A/B/N testing methods. This paper shares lessons l…
counterfactualCounterfactual InferenceExperimental Design