Leveraging User Simulation to Develop and Evaluate Conversational Information Access Agents
We observe a change in the way users access information, that is, the rise of conversational information access (CIA) agents. However, the automatic evaluation of these agents remains an open challenge. Moreover, the training of CIA agents is cumbersome as it mostly relies on conversational corpora, expert knowledge, and reinforcement learning. User simulation has been identified as a promising solution to tackle automatic evaluation and has been previously used in reinforcement learning. In this research, we investigate how user simulation can be leveraged in the context of CIA. We organize the work in three parts. We begin with the identification of requirements for user simulators for training and evaluating CIA agents and compare existing types of simulator regarding these. Then, we plan to combine these different types of simulators into a new hybrid simulator. Finally, we aim to extend simulators to handle more complex information seeking scenarios.
Code (0)
등록된 구현이 없습니다.
Tasks
Conversational Information AccessReinforcement LearningUser SimulationSimilar Papers 제목 키워드 기반
Towards a Formal Characterization of User Simulation Objectives in Conversational Information Access
User simulation is a promising approach for automatically training and evaluating conversational information access agents, enabling the generation of synthetic dialogues and facilitating reproducible experiments at scal…
Conversational Information AccessUser SimulationGoal Alignment in LLM-Based User Simulators for Conversational AI
User simulators are essential to conversational AI, enabling scalable agent development and evaluation through simulated interactions. While current Large Language Models (LLMs) have advanced user simulation capabilities…
Identifying Breakdowns in Conversational Recommender Systems using User Simulation
We present a methodology to systematically test conversational recommender systems with regards to conversational breakdowns. It involves examining conversations generated between the system and simulated users for a set…
Conversational RecommendationDiagnosticUser SimulationMulti-trait User Simulation with Adaptive Decoding for Conversational Task Assistants
Conversational systems must be robust to user interactions that naturally exhibit diverse conversational traits. Capturing and simulating these diverse traits coherently and efficiently presents a complex challenge. This…
DiversityUser SimulationEvaluating Conversational Recommender Systems via User Simulation
Conversational information access is an emerging research area. Currently, human evaluation is used for end-to-end system evaluation, which is both very time and resource intensive at scale, and thus becomes a bottleneck…
Conversational Information AccessRecommendation SystemsUser Simulation