Concept -- An Evaluation Protocol on Conversational Recommender Systems with System-centric and User-centric Factors
The conversational recommendation system (CRS) has been criticized regarding its user experience in real-world scenarios, despite recent significant progress achieved in academia. Existing evaluation protocols for CRS may prioritize system-centric factors such as effectiveness and fluency in conversation while neglecting user-centric aspects. Thus, we propose a new and inclusive evaluation protocol, Concept, which integrates both system- and user-centric factors. We conceptualise three key characteristics in representing such factors and further divide them into six primary abilities. To implement Concept, we adopt a LLM-based user simulator and evaluator with scoring rubrics that are tailored for each primary ability. Our protocol, Concept, serves a dual purpose. First, it provides an overview of the pros and cons in current CRS models. Second, it pinpoints the problem of low usability in the "omnipotent" ChatGPT and offers a comprehensive reference guide for evaluating CRS, thereby setting the foundation for CRS improvement.
Code (1)
Tasks
Conversational RecommendationRecommendation SystemsSimilar Papers 제목 키워드 기반
Evaluating Large Language Models as Generative User Simulators for Conversational Recommendation
Synthetic users are cost-effective proxies for real users in the evaluation of conversational recommender systems. Large language models show promise in simulating human-like behavior, raising the question of their abili…
Conversational RecommendationModel SelectionRecommendation SystemsUnderstanding User Intent Modeling for Conversational Recommender Systems: A Systematic Literature Review
Context: User intent modeling is a crucial process in Natural Language Processing that aims to identify the underlying purpose behind a user's request, enabling personalized responses. With a vast array of approaches int…
Model SelectionRecommendation SystemsSystematic Literature ReviewEvaluating Conversational Recommender Systems with Large Language Models: A User-Centric Evaluation Framework
Conversational recommender systems (CRS) involve both recommendation and dialogue tasks, which makes their evaluation a unique challenge. Although past research has analyzed various factors that may affect user satisfact…
Recommendation SystemsRethinking the Evaluation for Conversational Recommendation in the Era of Large Language Models
The recent success of large language models (LLMs) has shown great potential to develop more powerful conversational recommender systems (CRSs), which rely on natural language conversations to satisfy user needs. In this…
Conversational RecommendationExplanation GenerationRecommendation SystemsEvaluating Conversational Recommender Systems: A Landscape of Research
Conversational recommender systems aim to interactively support online users in their information search and decision-making processes in an intuitive way. With the latest advances in voice-controlled devices, natural la…
Decision MakingRecommendation Systems