RADio -- Rank-Aware Divergence Metrics to Measure Normative Diversity in News Recommendations
In traditional recommender system literature, diversity is often seen as the opposite of similarity, and typically defined as the distance between identified topics, categories or word models. However, this is not expressive of the social science's interpretation of diversity, which accounts for a news organization's norms and values and which we here refer to as normative diversity. We introduce RADio, a versatile metrics framework to evaluate recommendations according to these normative goals. RADio introduces a rank-aware Jensen Shannon (JS) divergence. This combination accounts for (i) a user's decreasing propensity to observe items further down a list and (ii) full distributional shifts as opposed to point estimates. We evaluate RADio's ability to reflect five normative concepts in news recommendations on the Microsoft News Dataset and six (neural) recommendation algorithms, with the help of our metadata enrichment pipeline. We find that RADio provides insightful estimates that can potentially be used to inform news recommender system design.
Code (0)
등록된 구현이 없습니다.
Tasks
DiversityRecommendation SystemsSimilar Papers 제목 키워드 기반
The Lovasz-Bregman Divergence and connections to rank aggregation, clustering, and web ranking
We extend the recently introduced theory of Lovasz-Bregman (LB) divergences (Iyer & Bilmes 2012) in several ways. We show that they represent a distortion between a "score" and an "ordering", thus providing a new view of…
ClusteringInformation RetrievalLearning-To-RankRetrievalThe Lovasz-Bregman Divergence and connections to rank aggregation, clustering, and web ranking
We extend the recently introduced theory of Lovasz-Bregman (LB) divergences (Iyer & Bilmes, 2012) in several ways. We show that they represent a distortion between a 'score' and an 'ordering', thus providing a new view o…
ClusteringInformation RetrievalLearning-To-RankRetrievalGraph-Aware Late Chunking for Retrieval-Augmented Generation in Biomedical Literature
Retrieval-Augmented Generation (RAG) systems for biomedical literature are typically evaluated using ranking metrics like Mean Reciprocal Rank (MRR), which measure how well the system identifies the single most relevant …
Boundary DetectionQuantifying Divergence in Inter-LLM Communication Through API Retrieval and Ranking
Large language models (LLMs) increasingly operate as autonomous agents that reason over external APIs to perform complex tasks. However, their reliability and agreement remain poorly characterized. We present a unified b…
Sentiment AnalysisOn Variants of Root Normalised Order-aware Divergence and a Divergence based on Kendall's Tau
This paper reports on a follow-up study of the work reported in Sakai, which explored suitable evaluation measures for ordinal quantification tasks. More specifically, the present study defines and evaluates, in addition…