Distributionally-Informed Recommender System Evaluation
Current practice for evaluating recommender systems typically focuses on point estimates of user-oriented effectiveness metrics or business metrics, sometimes combined with additional metrics for considerations such as diversity and novelty. In this paper, we argue for the need for researchers and practitioners to attend more closely to various distributions that arise from a recommender system (or other information access system) and the sources of uncertainty that lead to these distributions. One immediate implication of our argument is that both researchers and practitioners must report and examine more thoroughly the distribution of utility between and within different stakeholder groups. However, distributions of various forms arise in many more aspects of the recommender systems experimental process, and distributional thinking has substantial ramifications for how we design, evaluate, and present recommender systems evaluation and research results. Leveraging and emphasizing distributions in the evaluation of recommender systems is a necessary step to ensure that the systems provide appropriate and equitably-distributed benefit to the people they affect.
Code (0)
등록된 구현이 없습니다.
Tasks
DiversityRecommendation SystemsSimilar Papers 제목 키워드 기반
Fair Recommendations with Limited Sensitive Attributes: A Distributionally Robust Optimization Approach
As recommender systems are indispensable in various domains such as job searching and e-commerce, providing equitable recommendations to users with different sensitive attributes becomes an imperative requirement. Prior …
AttributeFairnessRecommendation SystemsDeviation-Based Learning: Training Recommender Systems Using Informed User Choice
This paper proposes a new approach to training recommender systems called deviation-based learning. The recommender and rational users have different knowledge. The recommender learns user knowledge by observing what act…
Recommendation SystemsEmpowering recommender systems using automatically generated Knowledge Graphs and Reinforcement Learning
Personalized recommender systems play a crucial role in direct marketing, particularly in financial services, where delivering relevant content can enhance customer engagement and promote informed decision-making. This s…
Decision MakingKnowledge GraphsManagementMarketing+3Fairness and Transparency in Recommendation: The Users' Perspective
Though recommender systems are defined by personalization, recent work has shown the importance of additional, beyond-accuracy objectives, such as fairness. Because users often expect their recommendations to be purely p…
FairnessRecommendation SystemsDebiasing Sequential Recommenders through Distributionally Robust Optimization over System Exposure
Sequential recommendation (SR) models are typically trained on user-item interactions which are affected by the system exposure bias, leading to the user preference learned from the biased SR model not being fully consis…
Sequential Recommendation