An Evaluation Protocol for Generative Conversational Systems
There is a multitude of novel generative models for open-domain conversational systems; however, there is no systematic evaluation of different systems. Systematic comparisons require consistency in experimental design, evaluation sets, conversational systems and their outputs, and statistical analysis. We lay out a protocol for the evaluation of conversational models using head-to-head pairwise comparison. We analyze ten recent models that claim state-of-the-art performance using a paired head-to-head performance (win-loss-tie) on five evaluation datasets. Our findings show that DialoGPT and Blender are superior systems using Bradley-Terry model and TrueSkill ranking methods. These findings demonstrate the feasibility of our protocol to evaluate conversational agents and evaluation sets. Finally, we make all code and evaluations publicly available for researchers to compare their model to other state-of-the-art dialog models.
Code (0)
등록된 구현이 없습니다.
Tasks
Experimental DesignMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Evaluating Large Language Models as Generative User Simulators for Conversational Recommendation
Synthetic users are cost-effective proxies for real users in the evaluation of conversational recommender systems. Large language models show promise in simulating human-like behavior, raising the question of their abili…
Conversational RecommendationModel SelectionRecommendation SystemsConcept -- An Evaluation Protocol on Conversational Recommender Systems with System-centric and User-centric Factors
The conversational recommendation system (CRS) has been criticized regarding its user experience in real-world scenarios, despite recent significant progress achieved in academia. Existing evaluation protocols for CRS ma…
Conversational RecommendationRecommendation SystemsRethinking the Evaluation for Conversational Recommendation in the Era of Large Language Models
The recent success of large language models (LLMs) has shown great potential to develop more powerful conversational recommender systems (CRSs), which rely on natural language conversations to satisfy user needs. In this…
Conversational RecommendationExplanation GenerationRecommendation SystemsTowards Unified Dialogue System Evaluation: A Comprehensive Analysis of Current Evaluation Protocols
As conversational AI-based dialogue management has increasingly become a trending topic, the need for a standardized and reliable evaluation procedure grows even more pressing. The current state of affairs suggests vario…
Dialogue ManagementManagementDMF: A Deterministic Memory Framework for Conversational AI Agents
Conversational AI agents require memory systems that are both scalable and semantically coherent across long interaction horizons. Existing approaches rely predominantly on large language model (LLM)-based summarisation …