Dialogue Evaluation with Offline Reinforcement Learning
Task-oriented dialogue systems aim to fulfill user goals through natural language interactions. They are ideally evaluated with human users, which however is unattainable to do at every iteration of the development phase. Simulated users could be an alternative, however their development is nontrivial. Therefore, researchers resort to offline metrics on existing human-human corpora, which are more practical and easily reproducible. They are unfortunately limited in reflecting real performance of dialogue systems. BLEU for instance is poorly correlated with human judgment, and existing corpus-based metrics such as success rate overlook dialogue context mismatches. There is still a need for a reliable metric for task-oriented systems with good generalization and strong correlation with human judgements. In this paper, we propose the use of offline reinforcement learning for dialogue evaluation based on a static corpus. Such an evaluator is typically called a critic and utilized for policy optimization. We go one step further and show that offline RL critics can be trained on a static corpus for any dialogue system as external evaluators, allowing dialogue performance comparisons across various types of systems. This approach has the benefit of being corpus- and model-independent, while attaining strong correlation with human judgements, which we confirm via an interactive user trial.
Code (0)
등록된 구현이 없습니다.
Tasks
Dialogue EvaluationOffline RLreinforcement-learningReinforcement LearningReinforcement Learning (RL)Task-Oriented Dialogue SystemsSimilar Papers 제목 키워드 기반
Building Persona Consistent Dialogue Agents with Offline Reinforcement Learning
Maintaining a consistent persona is a key quality for any open domain dialogue system. Current state-of-the-art systems do this by training agents with supervised learning or online reinforcement learning (RL). However, …
ChatbotOffline RLreinforcement-learningReinforcement Learning+1On the Effectiveness of Offline RL for Dialogue Response Generation
A common training technique for language models is teacher forcing (TF). TF attempts to match human language exactly, even though identical meanings can be expressed in different ways. This motivates use of sequence-leve…
Offline RLreinforcement-learningReinforcement Learning (RL)Response GenerationCHAI: A CHatbot AI for Task-Oriented Dialogue with Offline Reinforcement Learning
Conventionally, generation of natural language for dialogue agents may be viewed as a statistical learning problem: determine the patterns in human-provided data and generate appropriate responses with similar statistica…
ChatbotOffline RLreinforcement-learningReinforcement Learning+1Offline Reinforcement Learning for Large Scale Language Action Spaces
Training a task-oriented dialogue agent can be naturally formulated as offline reinforcement learning (RL) problem, where the agent aims to learn a conversational strategy to achieve user goals, only from a dialogue corp…
Language ModelingLanguage ModellingOffline RLreinforcement-learning+2UP-NRPA: User Portrait based Nested Rollout Policy Adaptation for Planning with Large Language Models in Goal-oriented Dialogue Systems
To address the challenge that current dialogue policy planning methods struggle to dynamically adapt to diverse user characteristics, this paper proposes a User Portrait based Nested Rollout Policy Adaptation (UP-NRPA) o…
Reinforcement Learning