Repeatability Corner Cases in Document Ranking: The Impact of Score Ties
Document ranking experiments should be repeatable. However, the interaction between multi-threaded indexing and score ties during retrieval may yield non-deterministic rankings, making repeatability not as trivial as one might imagine. In the context of the open-source Lucene search engine, score ties are broken by internal document ids, which are assigned at index time. Due to multi-threaded indexing, which makes experimentation with large modern document collections practical, internal document ids are not assigned consistently between different index instances of the same collection, and thus score ties are broken unpredictably. This short paper examines the effectiveness impact of such score ties, quantifying the variability that can be attributed to this phenomenon. The obvious solution to this non-determinism and to ensure repeatable document ranking is to break score ties using external collection document ids. This approach, however, comes with measurable efficiency costs due to the necessity of consulting external identifiers during query evaluation.
Code (0)
등록된 구현이 없습니다.
Tasks
Document RankingRetrievalSimilar Papers 제목 키워드 기반
Faster and better: a machine learning approach to corner detection
The repeatability and efficiency of a corner detector determines how likely it is to be useful in a real-world application. The repeatability is importand because the same scene viewed from different positions should yie…
BIG-bench Machine LearningFrom Optimizable to Interactable: Mixed Digital Twin-Empowered Testing of Vehicle-Infrastructure Cooperation Systems
Sufficient testing under corner cases is critical for the long-term operation of vehicle-infrastructure cooperation systems (VICS). However, existing corner-case generation methods are primarily AI-driven, and VICS testi…
Chord Angle Deviation using Tangent (CADT), an Efficient and Robust Contour-based Corner Detector
Detection of corner is the most essential process in a large number of computer vision and image processing applications. We have mentioned a number of popular contour-based corner detectors in our paper. Among all these…
An Efficient Single Chord-based Accumulation Technique (SCA) to Detect More Reliable Corners
Corner detection is a vital operation in numerous computer vision applications. The Chord-to-Point Distance Accumulation (CPDA) detector is recognized as the contour-based corner detector producing the lowest localizatio…
Monte Carlo dropout increases model repeatability
The integration of artificial intelligence into clinical workflows requires reliable and robust models. Among the main features of robustness is repeatability. Much attention is given to classification performance withou…
ClassificationDensity EstimationMedical Image Analysismodel