MIMICS-Duo: Offline & Online Evaluation of Search Clarification
Asking clarification questions is an active area of research; however, resources for training and evaluating search clarification methods are not sufficient. To address this issue, we describe MIMICS-Duo, a new freely available dataset of 306 search queries with multiple clarifications (a total of 1,034 query-clarification pairs). MIMICS-Duo contains fine-grained annotations on clarification questions and their candidate answers and enhances the existing MIMICS datasets by enabling multi-dimensional evaluation of search clarification methods, including online and offline evaluation. We conduct extensive analysis to demonstrate the relationship between offline and online search clarification datasets and outline several research directions enabled by MIMICS-Duo. We believe that this resource will help researchers better understand clarification in search.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Online and Offline Evaluation in Search Clarification
The effectiveness of clarification question models in engaging users within search systems is currently constrained, casting doubt on their overall usefulness. To improve the performance of these models, it is crucial to…
Information RetrievalRetrievalMIMICS: A Large-Scale Data Collection for Search Clarification
Search clarification has recently attracted much attention due to its applications in search engines. It has also been recognized as a major component in conversational information seeking systems. Despite its importance…
2kRanking Clarifying Questions Based on Predicted User Engagement
To improve online search results, clarification questions can be used to elucidate the information need of the user. This research aims to predict the user engagement with the clarification pane as an indicator of releva…
Curiosity by Design: An LLM-based Coding Assistant Asking Clarification Questions
Large Language Models (LLMs) are increasingly used as coding assistants. However, the ambiguity of the developer's prompt often leads to incorrect code generation, as current models struggle to infer user intent without …
Prompt EngineeringCode GenerationDiverse and Specific Clarification Question Generation with Keywords
Product descriptions on e-commerce websites often suffer from missing important aspects. Clarification question generation (CQGen) can be a promising approach to help alleviate the problem. Unlike traditional QGen assumi…
DiversityQuestion GenerationQuestion-GenerationSpecificity