Unbiased Loss Functions for Multilabel Classification with Missing Labels
This paper considers binary and multilabel classification problems in a setting where labels are missing independently and with a known rate. Missing labels are a ubiquitous phenomenon in extreme multi-label classification (XMC) tasks, such as matching Wikipedia articles to a small subset out of the hundreds of thousands of possible tags, where no human annotator can possibly check the validity of all the negative samples. For this reason, propensity-scored precision -- an unbiased estimate for precision-at-k under a known noise model -- has become one of the standard metrics in XMC. Few methods take this problem into account already during the training phase, and all are limited to loss functions that can be decomposed into a sum of contributions from each individual label. A typical approach to training is to reduce the multilabel problem into a series of binary or multiclass problems, and it has been shown that if the surrogate task should be consistent for optimizing recall, the resulting loss function is not decomposable over labels. Therefore, this paper derives the unique unbiased estimators for the different multilabel reductions, including the non-decomposable ones. These estimators suffer from increased variance and may lead to ill-posed optimization problems, which we address by switching to convex upper-bounds. The theoretical considerations are further supplemented by an experimental study showing that the switch to unbiased estimators significantly alters the bias-variance trade-off and may thus require stronger regularization, which in some cases can negate the benefits of unbiased estimation.
Code (0)
등록된 구현이 없습니다.
Tasks
ArticlesClassificationExtreme Multi-Label ClassificationMissing LabelsMulti-Label ClassificationMUlTI-LABEL-ClASSIFICATIONSimilar Papers 제목 키워드 기반
sigmoidF1: A Smooth F1 Score Surrogate Loss for Multilabel Classification
Multiclass multilabel classification is the task of attributing multiple labels to examples via predictions. Current models formulate a reduction of the multilabel setting into either multiple binary classifications or m…
ClassificationUnbiased Loss Functions for Extreme Classification With Missing Labels
The goal in extreme multi-label classification (XMC) is to tag an instance with a small subset of relevant labels from an extremely large set of possible labels. In addition to the computational burden arising from large…
ClassificationExtreme Multi-Label ClassificationGeneral ClassificationMissing Labels+3Analysis and Optimization of Loss Functions for Multiclass, Top-k, and Multilabel Classification
Top-k error is currently a popular performance measure on large scale image classification benchmarks such as ImageNet and Places. Despite its wide acceptance, our understanding of this metric is limited as most of the p…
General Classificationimage-classificationImage ClassificationOn Aggregation in Ensembles of Multilabel Classifiers
While a variety of ensemble methods for multilabel classification have been proposed in the literature, the question of how to aggregate the predictions of the individual members of the ensemble has received little atten…
General ClassificationMultilabel Classification using Bayesian Compressed Sensing
In this paper, we present a Bayesian framework for multilabel classification using compressed sensing. The key idea in compressed sensing for multilabel classification is to first project the label vector to a lower dime…
Active LearningClassificationcompressed sensingGeneral Classification+3