Disentangling Confidence Score Distribution for Out-of-Domain Intent Detection with Energy-Based Learning
Detecting Out-of-Domain (OOD) or unknown intents from user queries is essential in a task-oriented dialog system. Traditional softmax-based confidence scores are susceptible to the overconfidence issue. In this paper, we propose a simple but strong energy-based score function to detect OOD where the energy scores of OOD samples are higher than IND samples. Further, given a small set of labeled OOD samples, we introduce an energy-based margin objective for supervised OOD detection to explicitly distinguish OOD samples from INDs. Comprehensive experiments and analysis prove our method helps disentangle confidence score distributions of IND and OOD data.\footnote{Our code is available at \url{https://github.com/pris-nlp/EMNLP2022-energy_for_OOD/}.}
Code (0)
등록된 구현이 없습니다.
Tasks
Intent DetectionOut of Distribution (OOD) DetectionSimilar Papers 제목 키워드 기반
Intent-aware Recommendation via Disentangled Graph Contrastive Learning
Graph neural network (GNN) based recommender systems have become one of the mainstream trends due to the powerful learning ability from user behavior data. Understanding the user intents from behavior data is the key to …
Contrastive LearningGraph Neural NetworkRecommendation SystemsA Deep Generative Distance-Based Classifier for Out-of-Domain Detection with Mahalanobis Space
Detecting out-of-domain (OOD) input intents is critical in the task-oriented dialog system. Different from most existing methods that rely heavily on manually labeled OOD samples, we focus on the unsupervised OOD detecti…
Out of Distribution (OOD) DetectionAre Pretrained Transformers Robust in Intent Classification? A Missing Ingredient in Evaluation of Out-of-Scope Intent Detection
Pre-trained Transformer-based models were reported to be robust in intent classification. In this work, we first point out the importance of in-domain out-of-scope detection in few-shot intent recognition tasks and then …
intent-classificationIntent ClassificationIntent DetectionIntent RecognitionRevisit Overconfidence for OOD Detection: Reassigned Contrastive Learning with Adaptive Class-dependent Threshold
Detecting Out-of-Domain (OOD) or unknown intents from user queries is essential in a task-oriented dialog system. A key challenge of OOD detection is the overconfidence of neural models. In this paper, we comprehensively…
Contrastive LearningOut of Distribution (OOD) DetectionPaCoST: Paired Confidence Significance Testing for Benchmark Contamination Detection in Large Language Models
Large language models (LLMs) are known to be trained on vast amounts of data, which may unintentionally or intentionally include data from commonly used benchmarks. This inclusion can lead to cheatingly high scores on mo…