Quantifying the Uncertainty of Precision Estimates for Rule based Text Classifiers
Rule based classifiers that use the presence and absence of key sub-strings to make classification decisions have a natural mechanism for quantifying the uncertainty of their precision. For a binary classifier, the key insight is to treat partitions of the sub-string set induced by the documents as Bernoulli random variables. The mean value of each random variable is an estimate of the classifier's precision when presented with a document inducing that partition. These means can be compared, using standard statistical tests, to a desired or expected classifier precision. A set of binary classifiers can be combined into a single, multi-label classifier by an application of the Dempster-Shafer theory of evidence. The utility of this approach is demonstrated with a benchmark problem.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Quantifying Uncertainty in Natural Language Explanations of Large Language Models
Large Language Models (LLMs) are increasingly used as powerful tools for several high-stakes natural language processing (NLP) applications. Recent prompting works claim to elicit intermediate reasoning steps and key tok…
QUITE: Quantifying Uncertainty in Natural Language Text in Bayesian Reasoning Scenarios
Reasoning is key to many decision making processes. It requires consolidating a set of rule-like premises that are often associated with degrees of uncertainty and observations to draw conclusions. In this work, we addre…
Question AnsweringUncertainty estimation of machine learning spatial precipitation predictions from satellite data
Merging satellite and gauge data with machine learning produces high-resolution precipitation datasets, but uncertainty estimates are often missing. We addressed the gap of how to optimally provide such estimates by benc…
BenchmarkingFeature Importancequantile regressionregression+3Interval Deep Learning for Uncertainty Quantification in Safety Applications
Deep neural networks (DNNs) are becoming more prevalent in important safety-critical applications, where reliability in the prediction is paramount. Despite their exceptional prediction capabilities, current DNNs do not …
Deep LearningUncertainty QuantificationCompound Density Networks
Despite the huge success of deep neural networks (NNs), finding good mechanisms for quantifying their prediction uncertainty is still an open problem. It was recently shown, that using an ensemble of NNs trained with a p…
scoring rule