Uncertainty Estimation for the Open-Set Text Classification systems
Accurate uncertainty estimation is essential for building robust and trustworthy recognition systems. In this paper, we consider the open-set text classification (OSTC) task - and uncertainty estimation for it. For OSTC a text sample should be classified as one of the existing classes or rejected as unknown. To account for the different uncertainty types encountered in OSTC, we adapt the Holistic Uncertainty Estimation (HolUE) method for the text domain. Our approach addresses two major causes of prediction errors in text recognition systems: text uncertainty that stems from ill formulated queries and gallery uncertainty that is related the ambiguity of data distribution. By capturing these sources, it becomes possible to predict when the system will make a recognition error. We propose a new OSTC benchmark and conduct extensive experiments on a wide range of data, utilizing the authorship attribution, intent and topic classification datasets. HolUE achieves 40-365% improvement in Prediction Rejection Ratio (PRR) over the quality-based SCF baseline across datasets: 365% on Yahoo Answers (0.79 vs 0.17 at FPIR 0.1), 347% on DBPedia (0.85 vs 0.19), 240% on PAN authorship attribution (0.51 vs 0.15 at FPIR 0.5), and 40% on CLINC150 intent classification (0.73 vs~0.52). We make public our code and protocols https://github.com/Leonid-Erlygin/text_uncertainty.git
Code (0)
등록된 구현이 없습니다.
Tasks
Intent ClassificationText ClassificationSimilar Papers 제목 키워드 기반
CoMet: Context and Multiplicity Decomposition for Multimodal Uncertainty Estimation
Uncertainty estimation has been a long-standing challenge in AI models; it amounts to "knowing what you don't know," and metacognition is notoriously difficult even for humans (cf. the Dunning-Kruger effect). Although it…
Visual Question AnsweringAnswer GenerationHybrid uncertainty quantification for selective text classification in ambiguous tasks
Many text classification tasks are inherently ambiguous, which results in automatic systems having a high risk of making mistakes, in spite of using advanced machine learning models. For example, toxicity detection in us…
text-classificationText ClassificationUncertainty QuantificationTowards Principled Uncertainty Estimation for Deep Neural Networks
When the cost of misclassifying a sample is high, it is useful to have an accurate estimate of uncertainty in the prediction for that sample. There are also multiple types of uncertainty which are best estimated in diffe…
Bayesian InferenceContrastive Normalizing Flows for Uncertainty-Aware Parameter Estimation
Estimating physical parameters from data is a crucial application of machine learning (ML) in the physical sciences. However, systematic uncertainties, such as detector miscalibration, induce data distribution distortion…
parameter estimationRobust classificationUncertainty QuantificationE-NER: Evidential Deep Learning for Trustworthy Named Entity Recognition
Most named entity recognition (NER) systems focus on improving model performance, ignoring the need to quantify model uncertainty, which is critical to the reliability of NER systems in open environments. Evidential deep…
Deep Learningnamed-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)+1