Off-policy evaluation for learning-to-rank via interpolating the item-position model and the position-based model
A critical need for industrial recommender systems is the ability to evaluate recommendation policies offline, before deploying them to production. Unfortunately, widely used off-policy evaluation methods either make strong assumptions about how users behave that can lead to excessive bias, or they make fewer assumptions and suffer from large variance. We tackle this problem by developing a new estimator that mitigates the problems of the two most popular off-policy estimators for rankings, namely the position-based model and the item-position model. In particular, the new estimator, called INTERPOL, addresses the bias of a potentially misspecified position-based model, while providing an adaptable bias-variance trade-off compared to the item-position model. We provide theoretical arguments as well as empirical results that highlight the performance of our novel estimation approach.
Code (0)
등록된 구현이 없습니다.
Tasks
Learning-To-RankmodelOff-policy evaluationPositionRecommendation SystemsSimilar Papers 제목 키워드 기반
Unbiased Offline Evaluation for Learning to Rank with Business Rules
For industrial learning-to-rank (LTR) systems, it is common that the output of a ranking model is modified, either as a results of post-processing logic that enforces business requirements, or as a result of unforeseen d…
Learning-To-RankOff-policy evaluationTaking the Counterfactual Online: Efficient and Unbiased Online Evaluation for Ranking
Counterfactual evaluation can estimate Click-Through-Rate (CTR) differences between ranking systems based on historical interaction data, while mitigating the effect of position bias and item-selection bias. We introduce…
counterfactualPositionSelection biasTime-Constrained Recommendations: Reinforcement Learning Strategies for E-Commerce
Unlike traditional recommendation tasks, finite user time budgets introduce a critical resource constraint, requiring the recommender system to balance item relevance and evaluation cost. For example, in a mobile shoppin…
Reinforcement LearningDoubly Robust Off-Policy Evaluation for Ranking Policies under the Cascade Behavior Model
In real-world recommender systems and search engines, optimizing ranking decisions to present a ranked list of relevant items is critical. Off-policy evaluation (OPE) for ranking policies is thus gaining a growing intere…
Multi-Armed BanditsOff-policy evaluationRecommendation SystemsDecomposition and Interleaving for Variance Reduction of Post-click Metrics
In this study, we propose an efficient method for comparing the post-click metric (e.g., dwell time and conversion rate) of multiple rankings in online experiments. The proposed method involves (1) the decomposition of t…