Evaluating Conversational Recommender Systems with Large Language Models: A User-Centric Evaluation Framework
Conversational recommender systems (CRS) involve both recommendation and dialogue tasks, which makes their evaluation a unique challenge. Although past research has analyzed various factors that may affect user satisfaction with CRS interactions from the perspective of user studies, few evaluation metrics for CRS have been proposed. Recent studies have shown that LLMs can align with human preferences, and several LLM-based text quality evaluation measures have been introduced. However, the application of LLMs in CRS evaluation remains relatively limited. To address this research gap and advance the development of user-centric conversational recommender systems, this study proposes an automated LLM-based CRS evaluation framework, building upon existing research in human-computer interaction and psychology. The framework evaluates CRS from four dimensions: dialogue behavior, language expression, recommendation items, and response content. We use this framework to evaluate four different conversational recommender systems.
Code (0)
등록된 구현이 없습니다.
Tasks
Recommendation SystemsMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Evaluating Conversational Recommender Systems: A Landscape of Research
Conversational recommender systems aim to interactively support online users in their information search and decision-making processes in an intuitive way. With the latest advances in voice-controlled devices, natural la…
Decision MakingRecommendation SystemsUserSimCRS: A User Simulation Toolkit for Evaluating Conversational Recommender Systems
We present an extensible user simulation toolkit to facilitate automatic evaluation of conversational recommender systems. It builds on an established agenda-based approach and extends it with several novel elements, inc…
Recommendation SystemsText GenerationUser SimulationEvaluation of Synthetic Datasets for Conversational Recommender Systems
For researchers leveraging Large-Language Models (LLMs) in the generation of training datasets, especially for conversational recommender systems - the absence of robust evaluation frameworks has been a long-standing pro…
DiversityRecommendation SystemsUnderstanding User Intent Modeling for Conversational Recommender Systems: A Systematic Literature Review
Context: User intent modeling is a crucial process in Natural Language Processing that aims to identify the underlying purpose behind a user's request, enabling personalized responses. With a vast array of approaches int…
Model SelectionRecommendation SystemsSystematic Literature ReviewBehavior Alignment: A New Perspective of Evaluating LLM-based Conversational Recommender Systems
Large Language Models (LLMs) have demonstrated great potential in Conversational Recommender Systems (CRS). However, the application of LLMs to CRS has exposed a notable discrepancy in behavior between LLM-based CRS and …
Conversational RecommendationRecommendation Systems