Evaluating Uncertainty Calibration for Open-Set Recognition
Despite achieving enormous success in predictive accuracy for visual classification problems, deep neural networks (DNNs) suffer from providing overconfident probabilities on out-of-distribution (OOD) data. Yet, accurate uncertainty estimation is crucial for safe and reliable robot autonomy. In this paper, we evaluate popular calibration techniques for open-set conditions in a way that is distinctly different from the conventional evaluation of calibration methods on OOD data. Our results show that closed-set DNN calibration approaches are much less effective for open-set recognition, which highlights the need to develop new DNN calibration methods to address this problem.
Code (0)
등록된 구현이 없습니다.
Tasks
Open Set LearningSimilar Papers 제목 키워드 기반
Evaluating Predictive Uncertainty under Distributional Shift on Dialogue Dataset
In open-domain dialogues, predictive uncertainties are mainly evaluated in a domain shift setting to cope with out-of-distribution inputs. However, in real-world conversations, there could be more extensive distributiona…
Uncertainty in Action: Confidence Elicitation in Embodied Agents
Expressing confidence is challenging for embodied agents navigating dynamic multimodal environments, where uncertainty arises from both perception and decision-making processes. We present the first work investigating em…
Decision MakingMinecraftRevisiting Uncertainty Estimation and Calibration of Large Language Models
As large language models (LLMs) are increasingly deployed in high-stakes applications, robust uncertainty estimation is essential for ensuring the safe and trustworthy deployment of LLMs. We present the most comprehensiv…
Mixture-of-ExpertsMMLUQuantizationEnhancing Multi-view Open-set Learning via Ambiguity Uncertainty Calibration and View-wise Debiasing
Existing multi-view learning models struggle in open-set scenarios due to their implicit assumption of class completeness. Moreover, static view-induced biases, which arise from spurious view-label associations formed du…
Are Data Augmentation Methods in Named Entity Recognition Applicable for Uncertainty Estimation?
This work investigates the impact of data augmentation on confidence calibration and uncertainty estimation in Named Entity Recognition (NER) tasks. For the future advance of NER in safety-critical fields like healthcare…
Data Augmentationnamed-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)+1