Word-Level Uncertainty Estimation for Black-Box Text Classifiers using RNNs
Estimating uncertainties of Neural Network predictions paves the way towards more reliable and trustful text classifications. However, common uncertainty estimation approaches remain as black-boxes without explaining which features have led to the uncertainty of a prediction. This hinders users from understanding the cause of unreliable model behaviour. We introduce an approach to decompose and visualize the uncertainty of text classifiers at the level of words. Our approach builds on top of Recurrent Neural Networks and Bayesian modelling in order to provide detailed explanations of uncertainties, enabling a deeper reasoning about unreliable model behaviours. We conduct a preliminary experiment to check the impact and correctness of our approach. By explaining and investigating the predictive uncertainties of a sentiment analysis task, we argue that our approach is able to provide a more profound understanding of artificial decision making.
Code (1)
Tasks
Decision MakingSentiment AnalysisSimilar Papers 제목 키워드 기반
Font Identification in Historical Documents Using Active Learning
Identifying the type of font (e.g., Roman, Blackletter) used in historical documents can help optical character recognition (OCR) systems produce more accurate text transcriptions. Towards this end, we present an active-…
Active LearningDiversityOptical Character RecognitionOptical Character Recognition (OCR)BiG-SURE - Bipartite Graph for Semantic Uncertainty and Reliability Estimation of LLMs
Reliable uncertainty estimation is a crucial requirement for deploying large language models (LLMs) and vision-language models (VLMs) in safety-critical settings, especially when the model parameters are not accessible (…
Uncertainty Estimation and Quantification for LLMs: A Simple Supervised Approach
In this paper, we study the problem of uncertainty estimation and calibration for LLMs. We begin by formulating the uncertainty estimation problem, a relevant yet underexplored area in existing literature. We then propos…
Investigating the Helpfulness of Word-Level Quality Estimation for Post-Editing Machine Translation Output
Compared to fully manual translation, post-editing (PE) machine translation (MT) output can save time and reduce errors. Automatic word-level quality estimation (QE) aims to predict the correctness of words in MT output …
Machine TranslationTranslationUncertainty in Natural Language Processing: Sources, Quantification, and Applications
As a main field of artificial intelligence, natural language processing (NLP) has achieved remarkable success via deep neural networks. Plenty of NLP tasks have been addressed in a unified manner, with various tasks bein…
Uncertainty Quantification