Confidence Estimation Using Unlabeled Data
Overconfidence is a common issue for deep neural networks, limiting their deployment in real-world applications. To better estimate confidence, existing methods mostly focus on fully-supervised scenarios and rely on training labels. In this paper, we propose the first confidence estimation method for a semi-supervised setting, when most training labels are unavailable. We stipulate that even with limited training labels, we can still reasonably approximate the confidence of model on unlabeled samples by inspecting the prediction consistency through the training process. We use training consistency as a surrogate function and propose a consistency ranking loss for confidence estimation. On both image classification and segmentation tasks, our method achieves state-of-the-art performances in confidence estimation. Furthermore, we show the benefit of the proposed method through a downstream active learning task. The code is available at https://github.com/TopoXLab/consistency-ranking-loss
Code (1)
Tasks
Active Learningimage-classificationImage ClassificationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Semi-Supervised Partial Label Learning via Confidence-Rated Margin Maximization
Partial label learning assumes inaccurate supervision where each training example is associated with a set of candidate labels, among which only one is valid. In many real-world scenarios, however, it is costly and time-…
Partial Label LearningvalidPseudo-Loss Confidence Metric for Semi-Supervised Few-Shot Learning
Semi-supervised few-shot learning is developed to train a classifier that can adapt to new tasks with limited labeled data and a fixed quantity of unlabeled data. Most semi-supervised few-shot learning methods select…
Few-Shot LearningImportance Weight Estimation and Generalization in Domain Adaptation under Label Shift
We study generalization under labeled shift for categorical and general normed label spaces. We propose a series of methods to estimate the importance weights from labeled source to unlabeled target domain and provide co…
Domain AdaptationGeneralization BoundsOperator learningThe Impact of Unlabeled Patterns in Rademacher Complexity Theory for Kernel Classifiers
We derive here new generalization bounds, based on Rademacher Complexity theory, for model selection and error estimation of linear (kernel) classifiers, which exploit the availability of unlabeled samples. In parti…
Generalization BoundsModel SelectionDistantly Supervised Named Entity Recognition via Confidence-Based Multi-Class Positive and Unlabeled Learning
In this paper, we study the named entity recognition (NER) problem under distant supervision. Due to the incompleteness of the external dictionaries and/or knowledge bases, such distantly annotated training data usually …
named-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)NER