Some Theory For Practical Classifier Validation
We compare and contrast two approaches to validating a trained classifier while using all in-sample data for training. One is simultaneous validation over an organized set of hypotheses (SVOOSH), the well-known method that began with VC theory. The other is withhold and gap (WAG). WAG withholds a validation set, trains a holdout classifier on the remaining data, uses the validation data to validate that classifier, then adds the rate of disagreement between the holdout classifier and one trained using all in-sample data, which is an upper bound on the difference in error rates. We show that complex hypothesis classes and limited training data can make WAG a favorable alternative.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
On The Reasons Behind Decisions
Recent work has shown that some common machine learning classifiers can be compiled into Boolean circuits that have the same input-output behavior. We present a theory for unveiling the reasons behind the decisions made …
counterfactualDual Teaching: A Practical Semi-supervised Wrapper Method
Semi-supervised wrapper methods are concerned with building effective supervised classifiers from partially labeled data. Though previous works have succeeded in some fields, it is still difficult to apply semi-supervise…
On the Learnability of Out-of-distribution Detection
Supervised learning aims to train a classifier under the assumption that training and test data are from the same distribution. To ease the above assumption, researchers have studied a more realistic setting: out-of-dist…
Learning TheoryOut-of-Distribution DetectionOut of Distribution (OOD) DetectionPAC learningSPOCC: Scalable POssibilistic Classifier Combination -- toward robust aggregation of classifiers
We investigate a problem in which each member of a group of learners is trained separately to solve the same classification task. Each learner has access to a training dataset (possibly with overlap across learners) but …
Is Out-of-Distribution Detection Learnable?
Supervised learning aims to train a classifier under the assumption that training and test data are from the same distribution. To ease the above assumption, researchers have studied a more realistic setting: out-of-dist…
DiversityLearning TheoryOut-of-Distribution DetectionOut of Distribution (OOD) Detection+1