ADDIS: an adaptive discarding algorithm for online FDR control with conservative nulls
Major internet companies routinely perform tens of thousands of A/B tests each year. Such large-scale sequential experimentation has resulted in a recent spurt of new algorithms that can provably control the false discovery rate (FDR) in a fully online fashion. However, current state-of-the-art adaptive algorithms can suffer from a significant loss in power if null p-values are conservative (stochastically larger than the uniform distribution), a situation that occurs frequently in practice. In this work, we introduce a new adaptive discarding method called ADDIS that provably controls the FDR and achieves the best of both worlds: it enjoys appreciable power increase over all existing methods if nulls are conservative (the practical case), and rarely loses power if nulls are exactly uniformly distributed (the ideal case). We provide several practical insights on robust choices of tuning parameters, and extend the idea to asynchronous and offline settings as well.
Code (1)
Similar Papers 제목 키워드 기반
REBAR: Low-variance, unbiased gradient estimates for discrete latent variable models
Learning in models with discrete latent variables is challenging due to high variance gradient estimators. Generally, approaches have relied on control variates to reduce the variance of the REINFORCE estimator. Recent w…
Learning to Forget: Continual Learning with Adaptive Weight Decay
Continual learning agents with finite capacity must balance acquiring new knowledge with retaining the old. This requires controlled forgetting of knowledge that is no longer needed, freeing up capacity to learn. Weight …
Continual LearningAdaptive SpikeDeep-Classifier: Self-organizing and self-supervised machine learning algorithm for online spike sorting
Objective. Research on brain-computer interfaces (BCIs) is advancing towards rehabilitating severely disabled patients in the real world. Two key factors for successful decoding of user intentions are the size of implant…
channel selectionSpike Sorting1st AfricaNLP Workshop Proceedings, 2020
Proceedings of the 1st AfricaNLP Workshop held on 26th April alongside ICLR 2020, Virtual Conference, Formerly Addis Ababa Ethiopia.
Adaptive Matching Pursuit based Online Identification and Control Scheme for Nonlinear Systems
The complexity of adaptive control of nonlinear time-varying systems requires the use of novel methods that have lower computational complexity as well as ensuring good performance under time-varying parameter changes. I…