HINT3: Raising the bar for Intent Detection in the Wild
Intent Detection systems in the real world are exposed to complexities of imbalanced datasets containing varying perception of intent, unintended correlations and domain-specific aberrations. To facilitate benchmarking which can reflect near real-world scenarios, we introduce 3 new datasets created from live chatbots in diverse domains. Unlike most existing datasets that are crowdsourced, our datasets contain real user queries received by the chatbots and facilitates penalising unwanted correlations grasped during the training process. We evaluate 4 NLU platforms and a BERT based classifier and find that performance saturates at inadequate levels on test sets because all systems latch on to unintended patterns in training data.
Code (1)
Tasks
BenchmarkingIntent DetectionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Unknown Intent Detection Using Multi-Objective Optimization on Deep Learning Classifiers
Modelling and understanding dialogues in a conversation depends on identifying the user intent from the given text. Unknown or new intent detection is a critical task, as in a realistic scenario a user intent may frequen…
Intent DetectionIntent DiscoveryOnline Learning with Imperfect Hints
We consider a variant of the classical online linear optimization problem in which at every step, the online player receives a "hint" vector before choosing the action for that round. Rather surprisingly, it was shown th…
Multi-modal Facial Action Unit Detection with Large Pre-trained Models for the 5th Competition on Affective Behavior Analysis in-the-wild
Facial action unit detection has emerged as an important task within facial expression analysis, aimed at detecting specific pre-defined, objective facial expressions, such as lip tightening and cheek raising. This paper…
Action Unit DetectionFace AlignmentFacial Action Unit DetectionSuper-ResolutionLoD: Loss-difference OOD Detection by Intentionally Label-Noisifying Unlabeled Wild Data
Using unlabeled wild data containing both in-distribution (ID) and out-of-distribution (OOD) data to improve the safety and reliability of models has recently received increasing attention. Existing methods either design…
Social capital and resilience make an employee cooperate for coronavirus measures and lower his/her turnover intention
An important theme is how to maximize the cooperation of employees when dealing with crisis measures taken by the company. Therefore, to find out what kind of employees have cooperated with the company's measures in the …