On the Effect of Ranking Axioms on IR Evaluation Metrics
The study of IR evaluation metrics through axiomatic analysis enables a better understanding of their numerical properties. Some works have modelled the effectiveness of retrieval metrics with axioms that capture desirable properties on the set of rankings of documents. This paper formally explores the effect of these ranking axioms on the numerical values of some IR evaluation metrics. It focuses on the set of ranked lists of documents with multigrade relevance. The possible orderings in this set are derived from three commonly accepted ranking axioms on retrieval metrics; then, they are classified by their latticial properties. When relevant documents are prioritised, a subset of document rankings are identified: the join-irreducible elements, which have some resemblance to the concept of basis in vector space. It is possible to compute the precision, recall, RBP or DCG values of any ranking from their values in the join-irreducible elements. However this is not the case when the swapping of documents is considered.
Code (0)
등록된 구현이 없습니다.
Tasks
RetrievalSimilar Papers 제목 키워드 기반
On the Metric Properties of IR Evaluation Measures Based on Ranking Axioms
The axiomatic analysis of IR evaluation metrics has contributed to a better understanding of their properties. Some works have modelled the effectiveness of retrieval measures with axioms that capture desirable propertie…
RetrievalRankSHAP: Shapley Value Based Feature Attributions for Learning to Rank
Numerous works propose post-hoc, model-agnostic explanations for learning to rank, focusing on ordering entities by their relevance to a query through feature attribution methods. However, these attributions often weakly…
Computational EfficiencyLearning-To-RankElo Uncovered: Robustness and Best Practices in Language Model Evaluation
In Natural Language Processing (NLP), the Elo rating system, originally designed for ranking players in dynamic games such as chess, is increasingly being used to evaluate Large Language Models (LLMs) through "A vs B" pa…
Language Model EvaluationLanguage ModelingLanguage ModellingDesirable Rankings: A New Method for Ranking Outcomes of a Competitive Process
We consider the problem of aggregating individual preferences over alternatives into a social ranking. A key feature of the problems that we consider - and the one that allows us to obtain positive results, in contrast t…
An Axiomatic Approach to Regularizing Neural Ranking Models
Axiomatic information retrieval (IR) seeks a set of principle properties desirable in IR models. These properties when formally expressed provide guidance in the search for better relevance estimation functions. Neural r…
Information Retrievalparameter estimationRetrieval