User Simulation for Evaluating Information Access Systems
Information access systems, such as search engines, recommender systems, and conversational assistants, have become integral to our daily lives as they help us satisfy our information needs. However, evaluating the effectiveness of these systems presents a long-standing and complex scientific challenge. This challenge is rooted in the difficulty of assessing a system's overall effectiveness in assisting users to complete tasks through interactive support, and further exacerbated by the substantial variation in user behaviour and preferences. To address this challenge, user simulation emerges as a promising solution. This book focuses on providing a thorough understanding of user simulation techniques designed specifically for evaluation purposes. We begin with a background of information access system evaluation and explore the diverse applications of user simulation. Subsequently, we systematically review the major research progress in user simulation, covering both general frameworks for designing user simulators, utilizing user simulation for evaluation, and specific models and algorithms for simulating user interactions with search engines, recommender systems, and conversational assistants. Realizing that user simulation is an interdisciplinary research topic, whenever possible, we attempt to establish connections with related fields, including machine learning, dialogue systems, user modeling, and economics. We end the book with a detailed discussion of important future research directions, many of which extend beyond the evaluation of information access systems and are expected to have broader impact on how to evaluate interactive intelligent systems in general.
Code (0)
등록된 구현이 없습니다.
Tasks
Recommendation SystemsUser SimulationSimilar Papers 제목 키워드 기반
Evaluating Conversational Recommender Systems via User Simulation
Conversational information access is an emerging research area. Currently, human evaluation is used for end-to-end system evaluation, which is both very time and resource intensive at scale, and thus becomes a bottleneck…
Conversational Information AccessRecommendation SystemsUser SimulationLeveraging User Simulation to Develop and Evaluate Conversational Information Access Agents
We observe a change in the way users access information, that is, the rise of conversational information access (CIA) agents. However, the automatic evaluation of these agents remains an open challenge. Moreover, the tra…
Conversational Information AccessReinforcement LearningUser SimulationTowards a Formal Characterization of User Simulation Objectives in Conversational Information Access
User simulation is a promising approach for automatically training and evaluating conversational information access agents, enabling the generation of synthetic dialogues and facilitating reproducible experiments at scal…
Conversational Information AccessUser SimulationState of the Art of User Simulation approaches for conversational information retrieval
Conversational Information Retrieval (CIR) is an emerging field of Information Retrieval (IR) at the intersection of interactive IR and dialogue systems for open domain information needs. In order to optimize these inter…
Decision MakingInformation Retrievalreinforcement-learningReinforcement Learning+4Pessimistic Evaluation
Traditional evaluation of information access systems has focused primarily on average utility across a set of information needs (information retrieval) or users (recommender systems). In this work, we argue that evaluati…
FairnessInformation RetrievalRecommendation SystemsRetrieval