Online Evaluation Methods for the Causal Effect of Recommendations
Evaluating the causal effect of recommendations is an important objective because the causal effect on user interactions can directly leads to an increase in sales and user engagement. To select an optimal recommendation model, it is common to conduct A/B testing to compare model performance. However, A/B testing of causal effects requires a large number of users, making such experiments costly and risky. We therefore propose the first interleaving methods that can efficiently compare recommendation models in terms of causal effects. In contrast to conventional interleaving methods, we measure the outcomes of both items on an interleaved list and items not on the interleaved list, since the causal effect is the difference between outcomes with and without recommendations. To ensure that the evaluations are unbiased, we either select items with equal probability or weight the outcomes using inverse propensity scores. We then verify the unbiasedness and efficiency of online evaluation methods through simulated online experiments. The results indicate that our proposed methods are unbiased and that they have superior efficiency to A/B testing.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
The Informational Role of Online Recommendations: Evidence from a Field Experiment
We conduct a field experiment on a movie-recommendation platform to investigate whether and how online recommendations influence consumption choices. Using a within-subjects design, our experiment measures the causal eff…
Experimental DesignMovie RecommendationRecommendation SystemsTiming Process Interventions with Causal Inference and Reinforcement Learning
The shift from the understanding and prediction of processes to their optimization offers great benefits to businesses and other organizations. Precisely timed process interventions are the cornerstones of effective opti…
Causal Inferencereinforcement-learningReinforcement LearningReinforcement Learning (RL)Estimating Causal Effects with Double Machine Learning -- A Method Evaluation
The estimation of causal effects with observational data continues to be a very active research area. In recent years, researchers have developed new frameworks which use machine learning to relax classical assumptions n…
Causal Inference in Recommender Systems: A Survey of Strategies for Bias Mitigation, Explanation, and Generalization
In the era of information overload, recommender systems (RSs) have become an indispensable part of online service platforms. Traditional RSs estimate user interests and predict their future behaviors by utilizing correla…
Causal InferenceRecommendation SystemsCausality-Aware Neighborhood Methods for Recommender Systems
The business objectives of recommenders, such as increasing sales, are aligned with the causal effect of recommendations. Previous recommenders targeting for the causal effect employ the inverse propensity scoring (IPS) …
Causal InferenceRecommendation Systems