Probing Neural Dialog Models for Conversational Understanding
The predominant approach to open-domain dialog generation relies on end-to-end training of neural models on chat datasets. However, this approach provides little insight as to what these models learn (or do not learn) about engaging in dialog. In this study, we analyze the internal representations learned by neural open-domain dialog systems and evaluate the quality of these representations for learning basic conversational skills. Our results suggest that standard open-domain dialog systems struggle with answering questions, inferring contradiction, and determining the topic of conversation, among other tasks. We also find that the dyadic, turn-taking nature of dialog is not fully leveraged by these models. By exploring these limitations, we highlight the need for additional research into architectures and training methods that can better capture high-level information about dialog.
Code (1)
Tasks
Open-Domain DialogSimilar Papers 제목 키워드 기반
Lightweight Transformers for Conversational AI
To understand how training on conversational language impacts performance of pre-trained models on downstream dialogue tasks, we build compact Transformer-based Language Models from scratch on several large corpora of co…
GPUIntent DetectionNatural Language UnderstandingProSPer: Probing Human and Neural Network Language Model Understanding of Spatial Perspective
Understanding perspectival language is important for applications like dialogue systems and human-robot interaction. We propose a probe task that explores how well language models understand spatial perspective. We prese…
Language ModelingLanguage ModellingProbing the Robustness of Trained Metrics for Conversational Dialogue Systems
This paper introduces an adversarial method to stress-test trained metrics to evaluate conversational dialogue systems. The method leverages Reinforcement Learning to find response strategies that elicit optimal scores f…
reinforcement-learningReinforcement Learning (RL)Probing the Robustness of Trained Metrics for Conversational Dialogue Systems
This paper introduces an adversarial method to stress-test trained metrics for the evaluation of conversational dialogue systems. The method leverages Reinforcement Learning to find response strategies that elicit optima…
reinforcement-learningReinforcement Learning (RL)Dual Hierarchical Dialogue Policy Learning for Legal Inquisitive Conversational Agents
Most existing dialogue systems are user-driven, primarily designed to fulfill user requests. However, in many critical real-world scenarios, a conversational agent must proactively extract information to achieve its own …
Hierarchical Reinforcement Learning