Robustness Testing of Language Understanding in Task-Oriented Dialog
Most language understanding models in task-oriented dialog systems are trained on a small amount of annotated training data, and evaluated in a small set from the same distribution. However, these models can lead to system failure or undesirable output when being exposed to natural language perturbation or variation in practice. In this paper, we conduct comprehensive evaluation and analysis with respect to the robustness of natural language understanding models, and introduce three important aspects related to language understanding in real-world dialog systems, namely, language variety, speech characteristics, and noise perturbation. We propose a model-agnostic toolkit LAUG to approximate natural language perturbations for testing the robustness issues in task-oriented dialog. Four data augmentation approaches covering the three aspects are assembled in LAUG, which reveals critical robustness issues in state-of-the-art models. The augmented dataset through LAUG can be used to facilitate future research on the robustness testing of language understanding in task-oriented dialog.
Code (2)
Tasks
Data AugmentationNatural Language UnderstandingSimilar Papers 제목 키워드 기반
Enhancing LLM Robustness to Perturbed Instructions: An Empirical Study
Large Language Models (LLMs) are highly vulnerable to input perturbations, as even a small prompt change may result in a substantially different output. Existing methods to enhance LLM robustness are primarily focused on…
CoLADenoisingQNLISST-2A Simple Language Model for Task-Oriented Dialogue
Task-oriented dialogue is often decomposed into three tasks: understanding user input, deciding actions, and generating a response. While such decomposition might suggest a dedicated model for each sub-task, we find a si…
Dialogue State TrackingEnd-To-End Dialogue ModellingLanguage ModelingLanguage Modelling+4TAPE: Assessing Few-shot Russian Language Understanding
Recent advances in zero-shot and few-shot learning have shown promise for a scope of research and practical purposes. However, this fast-growing area lacks standardized evaluation suites for non-English languages, hinder…
Adversarial AttackAdversarial TextEthicsFew-Shot Learning+3Improving Robustness of Task Oriented Dialog Systems
Task oriented language understanding in dialog systems is often modeled using intents (task of a query) and slots (parameters for that task). Intent detection and slot tagging are, in turn, modeled using sentence classif…
Adversarial AttackData AugmentationIntent DetectionSentence+2Investigation of Language Understanding Impact for Reinforcement Learning Based Dialogue Systems
Language understanding is a key component in a spoken dialogue system. In this paper, we investigate how the language understanding module influences the dialogue system performance by conducting a series of systematic e…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)