Best of Three Worlds: Adaptive Experimentation for Digital Marketing in Practice
Adaptive experimental design (AED) methods are increasingly being used in industry as a tool to boost testing throughput or reduce experimentation cost relative to traditional A/B/N testing methods. However, the behavior and guarantees of such methods are not well-understood beyond idealized stationary settings. This paper shares lessons learned regarding the challenges of naively using AED systems in industrial settings where non-stationarity is prevalent, while also providing perspectives on the proper objectives and system specifications in such settings. We developed an AED framework for counterfactual inference based on these experiences, and tested it in a commercial environment.
Code (0)
등록된 구현이 없습니다.
Tasks
counterfactualCounterfactual InferenceExperimental DesignMarketingSimilar Papers 제목 키워드 기반
ADDIS: an adaptive discarding algorithm for online FDR control with conservative nulls
Major internet companies routinely perform tens of thousands of A/B tests each year. Such large-scale sequential experimentation has resulted in a recent spurt of new algorithms that can provably control the false discov…
Active Learning with Logged Data
We consider active learning with logged data, where labeled examples are drawn conditioned on a predetermined logging policy, and the goal is to learn a classifier on the entire population, not just conditioned on the lo…
Active LearningA Best-of-both-worlds Algorithm for Bandits with Delayed Feedback with Robustness to Excessive Delays
We propose a new best-of-both-worlds algorithm for bandits with variably delayed feedback. In contrast to prior work, which required prior knowledge of the maximal delay $d_{\mathrm{max}}$ and had a linear dependence of …
Decision MakingThe Seven Worlds and Experiences of the Wireless Metaverse: Challenges and Opportunities
The wireless metaverse will create diverse user experiences at the intersection of the physical, digital, and virtual worlds. These experiences will enable novel interactions between the constituents (e.g., extended real…
Adaptive Experimentation with Delayed Binary Feedback
Conducting experiments with objectives that take significant delays to materialize (e.g. conversions, add-to-cart events, etc.) is challenging. Although the classical "split sample testing" is still valid for the delayed…
Multi-Armed Banditsvalid