Evaluating Generative Ad Hoc Information Retrieval
Recent advances in large language models have enabled the development of viable generative retrieval systems. Instead of a traditional document ranking, generative retrieval systems often directly return a grounded generated text as a response to a query. Quantifying the utility of the textual responses is essential for appropriately evaluating such generative ad hoc retrieval. Yet, the established evaluation methodology for ranking-based ad hoc retrieval is not suited for the reliable and reproducible evaluation of generated responses. To lay a foundation for developing new evaluation methods for generative retrieval systems, we survey the relevant literature from the fields of information retrieval and natural language processing, identify search tasks and system architectures in generative retrieval, develop a new user model, and study its operationalization.
Code (0)
등록된 구현이 없습니다.
Tasks
Ad-Hoc Information RetrievalDocument RankingInformation RetrievalRetrievalMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Evaluating Robustness of Generative Search Engine on Adversarial Factual Questions
Generative search engines have the potential to transform how people seek information online, but generated responses from existing large language models (LLMs)-backed generative search engines may not always be accurate…
RetrievalRetrieval-augmented GenerationHow Does Generative Retrieval Scale to Millions of Passages?
Popularized by the Differentiable Search Index, the emerging paradigm of generative retrieval re-frames the classic information retrieval problem into a sequence-to-sequence modeling task, forgoing external indices and e…
Information RetrievalPassage RankingRetrievalEvaluating a Generative Adversarial Framework for Information Retrieval
Recent advances in Generative Adversarial Networks (GANs) have resulted in its widespread applications to multiple domains. A recent model, IRGAN, applies this framework to Information Retrieval (IR) and has gained signi…
Information RetrievalRetrievalCoverageBench: Evaluating Information Coverage across Tasks and Domains
We wish to measure the information coverage of an ad hoc retrieval algorithm, that is, how much of the range of available relevant information is covered by the search results. Information coverage is a central aspect fo…
Generative Information Retrieval Evaluation
In this chapter, we consider generative information retrieval evaluation from two distinct but interrelated perspectives. First, large language models (LLMs) themselves are rapidly becoming tools for evaluation, with cur…
Information RetrievalRAGRetrievalRetrieval-augmented Generation