Augmented Test Collections: A Step in the Right Direction
In this position paper we argue that certain aspects of relevance assessment in the evaluation of IR systems are oversimplified and that human assessments represented by qrels should be augmented to take account of contextual factors and the subjectivity of the task at hand. We propose enhancing test collections used in evaluation with information related to human assessors and their interpretation of the task. Such augmented collections would provide a more realistic and user-focused evaluation, enabling us to better understand the evaluation process, the performance of systems and user interactions. A first step is to conduct user studies to examine in more detail what people actually do when we ask them to judge the relevance of a document.
Code (0)
등록된 구현이 없습니다.
Tasks
PositionSimilar Papers 제목 키워드 기반
REANIMATOR: Reanimate Retrieval Test Collections with Extracted and Synthetic Resources
Retrieval test collections are essential for evaluating information retrieval systems, yet they often lack generalizability across tasks. To overcome this limitation, we introduce REANIMATOR, a versatile framework design…
Information RetrievalRetrievalRetrieval-augmented GenerationLightFormer: An End-to-End Model for Intersection Right-of-Way Recognition Using Traffic Light Signals and an Attention Mechanism
For smart vehicles driving through signalised intersections, it is crucial to determine whether the vehicle has right of way given the state of the traffic lights. To address this issue, camera based sensors can be used …
A signal discovery step in interstellar communication
Prior work using synchronized, geographically spaced radio telescopes, and a radio interferometer, suggests that narrow-bandwidth polarized pulse pair measurements repeatedly falsify a noise-cause hypothesis, given a pri…
Discovering Causal Signals in Images
This paper establishes the existence of observable footprints that reveal the "causal dispositions" of the object categories appearing in collections of images. We achieve this goal in two steps. First, we take a learnin…
Causal DiscoveryIncorporating Legal Structure in Retrieval-Augmented Generation: A Case Study on Copyright Fair Use
This paper presents a domain-specific implementation of Retrieval-Augmented Generation (RAG) tailored to the Fair Use Doctrine in U.S. copyright law. Motivated by the increasing prevalence of DMCA takedowns and the lack …
Knowledge GraphsLegal ReasoningRAGRetrieval+1