A Rademacher Complexity Based Method fo rControlling Power and Confidence Level in Adaptive Statistical Analysis
While standard statistical inference techniques and machine learning generalization bounds assume that tests are run on data selected independently of the hypotheses, practical data analysis and machine learning are usually iterative and adaptive processes where the same holdout data is often used for testing a sequence of hypotheses (or models), which may each depend on the outcome of the previous tests on the same data. In this work, we present RadaBound a rigorous, efficient and practical procedure for controlling the generalization error when using a holdout sample for multiple adaptive testing. Our solution is based on a new application of the Rademacher Complexity generalization bounds, adapted to dependent tests. We demonstrate the statistical power and practicality of our method through extensive simulations and comparisons to alternative approaches.
Code (0)
등록된 구현이 없습니다.
Tasks
BIG-bench Machine LearningGeneralization BoundsSimilar Papers 제목 키워드 기반
Rademacher Complexity for Adversarially Robust Generalization
Many machine learning models are vulnerable to adversarial attacks; for example, adding adversarial perturbations that are imperceptible to humans can often make machine learning models produce wrong predictions with hig…
BIG-bench Machine LearningLocal Rademacher Complexity Bounds based on Covering Numbers
This paper provides a general result on controlling local Rademacher complexities, which captures in an elegant form to relate the complexities with constraint on the expected norm to the corresponding ones with constrai…
The Impact of Unlabeled Patterns in Rademacher Complexity Theory for Kernel Classifiers
We derive here new generalization bounds, based on Rademacher Complexity theory, for model selection and error estimation of linear (kernel) classifiers, which exploit the availability of unlabeled samples. In parti…
Generalization BoundsModel SelectionHuman Rademacher Complexity
We propose to use Rademacher complexity, originally developed in computational learning theory, as a measure of human learning capacity. Rademacher complexity measures a learners ability to fit random data, and can be u…
Generalization BoundsLearning TheoryRademacher Complexity Bounds for Non-I.I.D. Processes
This paper presents the first data-dependent generalization bounds for non-i.i.d. settings based on the notion of Rademacher complexity. Our bounds extend to the non-i.i.d. case existing Rademacher complexity bounds deri…
Generalization Bounds