Active Estimation of F-Measures
We address the problem of estimating the F-measure of a given model as accurately as possible on a fixed labeling budget. This problem occurs whenever an estimate cannot be obtained from held-out training data; for instance, when data that have been used to train the model are held back for reasons of privacy or do not reflect the test distribution. In this case, new test instances have to be drawn and labeled at a cost. An active estimation procedure selects instances according to an instrumental sampling distribution. An analysis of the sources of estimation error leads to an optimal sampling distribution that minimizes estimator variance. We explore conditions under which active estimates of F-measures are more accurate than estimates based on instances sampled from the test distribution.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Active Sampling of Multiple Sources for Sequential Estimation
Consider $K$ processes, each generating a sequence of identical and independent random variables. The probability measures of these processes have random parameters that must be estimated. Specifically, they share a para…
Active Learning for Derivative-Based Global Sensitivity Analysis with Gaussian Processes
We consider the problem of active learning for global sensitivity analysis of expensive black-box functions. Our aim is to efficiently learn the importance of different input variables, e.g., in vehicle safety experiment…
Active LearningGaussian ProcessesSensitivityThe Statistical Complexity of Interactive Decision Making
A fundamental challenge in interactive learning and decision making, ranging from bandit problems to reinforcement learning, is to provide sample-efficient, adaptive learning algorithms that achieve near-optimal regret. …
Decision Makingreinforcement-learningReinforcement Learning (RL)Model-Centric and Data-Centric Aspects of Active Learning for Deep Neural Networks
We study different aspects of active learning with deep neural networks in a consistent and unified way. i) We investigate incremental and cumulative training modes which specify how the newly labeled data are used for t…
Active LearningInformativenessAnalytic Mutual Information in Bayesian Neural Networks
Bayesian neural networks have successfully designed and optimized a robust neural network model in many application problems, including uncertainty quantification. However, with its recent success, information-theoretic …
Active Learningparameter estimationUncertainty Quantification