What Went Wrong? Explaining Overall Dialogue Quality through Utterance-Level Impacts
Improving user experience of a dialogue system often requires intensive developer effort to read conversation logs, run statistical analyses, and intuit the relative importance of system shortcomings. This paper presents a novel approach to automated analysis of conversation logs that learns the relationship between user-system interactions and overall dialogue quality. Unlike prior work on utterance-level quality prediction, our approach learns the impact of each interaction from the overall user rating without utterance-level annotation, allowing resultant model conclusions to be derived on the basis of empirical evidence and at low cost. Our model identifies interactions that have a strong correlation with the overall dialogue quality in a chatbot setting. Experiments show that the automated analysis from our model agrees with expert judgments, making this work the first to show that such weakly-supervised learning of utterance-level quality prediction is highly achievable.
Code (0)
등록된 구현이 없습니다.
Tasks
ChatbotWeakly-supervised LearningSimilar Papers 제목 키워드 기반
Visualizing and Understanding Atari Agents
While deep reinforcement learning (deep RL) agents are effective at maximizing rewards, it is often unclear what strategies they use to do so. In this paper, we take a step toward explaining deep RL agents through a case…
Deep Reinforcement LearningReinforcement LearningReinforcement Learning (RL)Analyzing the Effectiveness of Large Language Models on Text-to-SQL Synthesis
This study investigates various approaches to using Large Language Models (LLMs) for Text-to-SQL program synthesis, focusing on the outcomes and insights derived. Employing the popular Text-to-SQL dataset, spider, the go…
16kProgram SynthesisSQL SynthesisText to SQL+1Prediction of Video Game Development Problems Based on Postmortems using Different Word Embedding Techniques
The interactive entertainment industry is being actively involved with the development, marketing and sale of video games in the past decade. The increasing interest in video games has led to an increase in video game de…
MarketingSchema-Guided Dialogue State Tracking Task at DSTC8
This paper gives an overview of the Schema-Guided Dialogue State Tracking task of the 8th Dialogue System Technology Challenge. The goal of this task is to develop dialogue state tracking models suitable for large-scale …
Data AugmentationDialogue State TrackingZero-shot GeneralizationCounterfactual Control for Free from Generative Models
We introduce a method by which a generative model learning the joint distribution between actions and future states can be used to automatically infer a control scheme for any desired reward function, which may be altere…
counterfactual