RankDCG: Rank-Ordering Evaluation Measure
Ranking is used for a wide array of problems, most notably information retrieval (search). Kendall{'}s Ï„, Average Precision, and nDCG are a few popular approaches to the evaluation of ranking. When dealing with problems such as user ranking or recommendation systems, all these measures suffer from various problems, including the inability to deal with elements of the same rank, inconsistent and ambiguous lower bound scores, and an inappropriate cost function. We propose a new measure, a modification of the popular nDCG algorithm, named rankDCG, that addresses these problems. We provide a number of criteria for any effective ranking algorithm and show that only rankDCG satisfies them all. Results are presented on constructed and real data sets. We release a publicly available rankDCG evaluation package.
Code (0)
등록된 구현이 없습니다.
Tasks
Information RetrievalRecommendation SystemsRetrievalSimilar Papers 제목 키워드 기반
On the Metric Properties of IR Evaluation Measures Based on Ranking Axioms
The axiomatic analysis of IR evaluation metrics has contributed to a better understanding of their properties. Some works have modelled the effectiveness of retrieval measures with axioms that capture desirable propertie…
RetrievalAssessing top-$k$ preferences
Assessors make preference judgments faster and more consistently than graded judgments. Preference judgments can also recognize distinctions between items that appear equivalent under graded judgments. Unfortunately, pre…
Aristotelian Virtue Profiling of LLMs through Ethical Dilemmas
Large Language Models (LLMs) often face ethical tradeoffs in which several responses may be defensible but express different priorities, such as fairness, honesty, courage, or restraint. We introduce VirtueMap, a framewo…
Traits of a Leader: User Influence Level Prediction through Sociolinguistic Modeling
Recognition of a user's influence level has attracted much attention as human interactions move online. Influential users have the ability to sway others' opinions to achieve some goals. As a result, predicting users' le…
MisinformationMeasuring Calibration in Deep Learning
Overconfidence and underconfidence in machine learning classifiers is measured by calibration: the degree to which the probabilities predicted for each class match the accuracy of the classifier on that prediction. How o…
Deep Learning