Towards a Metric for Automated Conversational Dialogue System Evaluation and Improvement
We present "AutoJudge", an automated evaluation method for conversational dialogue systems. The method works by first generating dialogues based on self-talk, i.e. dialogue systems talking to itself. Then, it uses human ratings on these dialogues to train an automated judgement model. Our experiments show that AutoJudge correlates well with the human ratings and can be used to automatically evaluate dialogue systems, even in deployed systems. In a second part, we attempt to apply AutoJudge to improve existing systems. This works well for re-ranking a set of candidate utterances. However, our experiments show that AutoJudge cannot be applied as reward for reinforcement learning, although the metric can distinguish good from bad dialogues. We discuss potential reasons, but state here already that this is still an open question for further research.
Code (0)
등록된 구현이 없습니다.
Tasks
Open-Ended Question AnsweringReinforcement LearningReinforcement Learning (RL)Re-RankingSimilar Papers 제목 키워드 기반
Measuring Conversational Fluidity in Automated Dialogue Agents
We present an automated evaluation method to measure fluidity in conversational dialogue systems. The method combines various state of the art Natural Language tools into a classifier, and human ratings on these dialogue…
Evaluating Conversational Recommender Systems with Large Language Models: A User-Centric Evaluation Framework
Conversational recommender systems (CRS) involve both recommendation and dialogue tasks, which makes their evaluation a unique challenge. Although past research has analyzed various factors that may affect user satisfact…
Recommendation SystemsYour Students Don't Use LLMs Like You Wish They Did
Educational NLP systems are typically evaluated using engagement metrics and satisfaction surveys, which are at best a proxy for meeting pedagogical goals. We introduce six computational metrics for automated evaluation …
Dialogue EvaluationTowards Automatic Evaluation of Task-Oriented Dialogue Flows
Task-oriented dialogue systems rely on predefined conversation schemes (dialogue flows) often represented as directed acyclic graphs. These flows can be manually designed or automatically generated from previously record…
Task-Oriented Dialogue SystemsLEEETs-Dial: Linguistic Entrainment in End-to-End Task-oriented Dialogue systems
Linguistic entrainment, or alignment, represents a phenomenon where linguistic patterns employed by conversational participants converge to one another. While entrainment has been shown to produce a more natural user exp…
Task-Oriented Dialogue Systems