UserSimCRS: A User Simulation Toolkit for Evaluating Conversational Recommender Systems
We present an extensible user simulation toolkit to facilitate automatic evaluation of conversational recommender systems. It builds on an established agenda-based approach and extends it with several novel elements, including user satisfaction prediction, persona and context modeling, and conditional natural language generation. We showcase the toolkit with a pre-existing movie recommender system and demonstrate its ability to simulate dialogues that mimic real conversations, while requiring only a handful of manually annotated dialogues as training data.
Code (1)
Tasks
Recommendation SystemsText GenerationUser SimulationSimilar Papers 제목 키워드 기반
Lexara: A User-Centered Toolkit for Evaluating Large Language Models for Conversational Visual Analytics
Large Language Models (LLMs) are transforming Conversational Visual Analytics (CVA) by enabling data analysis through natural language. However, evaluating LLMs for CVA remains a challenge: requiring programming expertis…
SDialog: A Python Toolkit for End-to-End Agent Building, User Simulation, Dialog Generation, and Evaluation
We present SDialog, an MIT-licensed open-source Python toolkit that unifies dialog generation, evaluation and mechanistic interpretability into a single end-to-end framework for building and analyzing LLM-based conversat…
Audio GenerationADVISER: A Toolkit for Developing Multi-modal, Multi-domain and Socially-engaged Conversational Agents
We present ADVISER - an open-source, multi-domain dialog system toolkit that enables the development of multi-modal (incorporating speech, text and vision), socially-engaged (e.g. emotion recognition, engagement level pr…
BIG-bench Machine LearningEmotion RecognitionRecWizard: A Toolkit for Conversational Recommendation with Modular, Portable Models and Interactive User Interface
We present a new Python toolkit called RecWizard for Conversational Recommender Systems (CRS). RecWizard offers support for development of models and interactive user interface, drawing from the best practices of the Hug…
Conversational RecommendationRecommendation SystemsEvaluating Conversational Recommender Systems via User Simulation
Conversational information access is an emerging research area. Currently, human evaluation is used for end-to-end system evaluation, which is both very time and resource intensive at scale, and thus becomes a bottleneck…
Conversational Information AccessRecommendation SystemsUser Simulation