Stopping criterion for active learning based on deterministic generalization bounds
Active learning is a framework in which the learning machine can select the samples to be used for training. This technique is promising, particularly when the cost of data acquisition and labeling is high. In active learning, determining the timing at which learning should be stopped is a critical issue. In this study, we propose a criterion for automatically stopping active learning. The proposed stopping criterion is based on the difference in the expected generalization errors and hypothesis testing. We derive a novel upper bound for the difference in expected generalization errors before and after obtaining a new training datum based on PAC-Bayesian theory. Unlike ordinary PAC-Bayesian bounds, though, the proposed bound is deterministic; hence, there is no uncontrollable trade-off between the confidence and tightness of the inequality. We combine the upper bound with a statistical test to derive a stopping criterion for active learning. We demonstrate the effectiveness of the proposed method via experiments with both artificial and real datasets.
Code (0)
등록된 구현이 없습니다.
Tasks
Active LearningGeneralization BoundsTwo-sample testingSimilar Papers 제목 키워드 기반
Stopping Criterion for Active Learning Based on Error Stability
Active learning is a framework for supervised learning to improve the predictive performance by adaptively annotating a small number of samples. To realize efficient active learning, both an acquisition function that det…
Active LearningRisk-Aware Active Inverse Reinforcement Learning
Active learning from demonstration allows a robot to query a human for specific types of input to achieve efficient learning. Existing work has explored a variety of active query strategies; however, to our knowledge, no…
Active Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)Bounding the expected run-time of nonconvex optimization with early stopping
This work examines the convergence of stochastic gradient-based optimization algorithms that use early stopping based on a validation function. The form of early stopping we consider is that optimization terminates when …
Estimating the Success of Unsupervised Image to Image Translation
While in supervised learning, the validation error is an unbiased estimator of the generalization (test) error and complexity-based generalization bounds are abundant, no such bounds exist for learning a mapping in an un…
Generalization BoundsImage-to-Image TranslationTranslationUnsupervised Image-To-Image TranslationGeneralization Bounds and Stopping Rules for Learning with Self-Selected Data
Many learning paradigms self-select training data in light of previously learned parameters. Examples include active learning, semi-supervised learning, bandits, or boosting. Rodemann et al. (2024) unify them under the f…
Active LearningGeneralization Boundsvalid