Reaching the End of Unbiasedness: Uncovering Implicit Limitations of Click-Based Learning to Rank
Click-based learning to rank (LTR) tackles the mismatch between click frequencies on items and their actual relevance. The approach of previous work has been to assume a model of click behavior and to subsequently introduce a method for unbiasedly estimating preferences under that assumed model. The success of this approach is evident in that unbiased methods have been found for an increasing number of behavior models and types of bias. This work aims to uncover the implicit limitations of the high-level prevalent approach in the counterfactual LTR field. Thus, in contrast with limitations that follow from explicit assumptions, our aim is to recognize limitations that the field is currently unaware of. We do this by inverting the existing approach: we start by capturing existing methods in generic terms, and subsequently, from these generic descriptions we derive the click behavior for which these methods can be unbiased. Our inverted approach reveals that there are indeed implicit limitations to the counterfactual LTR approach: we find counterfactual estimation can only produce unbiased methods for click behavior based on affine transformations. In addition, we also recognize previously undiscussed limitations of click-modelling and pairwise approaches to click-based LTR. Our findings reveal that it is impossible for existing approaches to provide unbiasedness guarantees for all plausible click behavior models.
Code (0)
등록된 구현이 없습니다.
Tasks
counterfactualLearning-To-RankSimilar Papers 제목 키워드 기반
Entire-Space Variational Information Exploitation for Post-Click Conversion Rate Prediction
In recommender systems, post-click conversion rate (CVR) estimation is an essential task to model user preferences for items and estimate the value of recommendations. Sample selection bias (SSB) and data sparsity (DS) a…
Knowledge DistillationRecommendation SystemsSelection biasULTRA: An Unbiased Learning To Rank Algorithm Toolbox
Learning to rank systems has become an important aspect of our daily life. However, the implicit user feedback that is used to train many learning to rank models is usually noisy and suffered from user bias (i.e., positi…
Learning-To-RankMTRec: Learning to Align with User Preferences via Mental Reward Models
Recommendation models are predominantly trained using implicit user feedback, since explicit feedback is often costly to obtain. However, implicit feedback, such as clicks, does not always reflect users' real preferences…
Sequential RecommendationReinforcement LearningDoubly Robust Estimation of Causal Effect on CVR with Targeted Regularization
Post-click conversion rate (CVR) is a key metric in various scenarios including e-commerce and advertising, reflecting the efficiency and user experience in the second stage of the conversion process. Estimating the caus…
Causal InferenceDoubly-Robust Estimation for Correcting Position-Bias in Click Feedback for Unbiased Learning to Rank
Clicks on rankings suffer from position-bias: generally items on lower ranks are less likely to be examined - and thus clicked - by users, in spite of their actual preferences between items. The prevalent approach to unb…
counterfactualGeneral Reinforcement LearningLearning-To-RankPosition