Optimizing Long-term Value for Auction-Based Recommender Systems via On-Policy Reinforcement Learning
Auction-based recommender systems are prevalent in online advertising platforms, but they are typically optimized to allocate recommendation slots based on immediate expected return metrics, neglecting the downstream effects of recommendations on user behavior. In this study, we employ reinforcement learning to optimize for long-term return metrics in an auction-based recommender system. Utilizing temporal difference learning, a fundamental reinforcement learning algorithm, we implement an one-step policy improvement approach that biases the system towards recommendations with higher long-term user engagement metrics. This optimizes value over long horizons while maintaining compatibility with the auction framework. Our approach is grounded in dynamic programming ideas which show that our method provably improves upon the existing auction-based base policy. Through an online A/B test conducted on an auction-based recommender system which handles billions of impressions and users daily, we empirically establish that our proposed method outperforms the current production system in terms of long-term user engagement metrics.
Code (0)
등록된 구현이 없습니다.
Tasks
Recommendation Systemsreinforcement-learningReinforcement LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
PrefRec: Recommender Systems with Human Preferences for Reinforcing Long-term User Engagement
Current advances in recommender systems have been remarkably successful in optimizing immediate engagement. However, long-term user engagement, a more desirable performance metric, remains difficult to improve. Meanwhile…
Recommendation SystemsReinforcement Learning (RL)User Response in Ad Auctions: An MDP Formulation of Long-Term Revenue Optimization
We propose a new Markov Decision Process (MDP) model for ad auctions to capture the user response to the quality of ads, with the objective of maximizing the long-term discounted revenue. By incorporating user response, …
Optimizing Audio Recommendations for the Long-Term: A Reinforcement Learning Perspective
We present a novel podcast recommender system deployed at industrial scale. This system successfully optimizes personal listening journeys that unfold over months for hundreds of millions of listeners. In deviating from …
Recommendation Systemsreinforcement-learningReinforcement Learning (RL)Should I send this notification? Optimizing push notifications decision making by modeling the future
Most recommender systems are myopic, that is they optimize based on the immediate response of the user. This may be misaligned with the true objective, such as creating long term user satisfaction. In this work we focus …
Decision MakingModel-based Reinforcement LearningRecommendation SystemsReinforcement Learning (RL)Robust Pricing in Dynamic Mechanism Design
Motivated by the repeated sale of online ads via auctions, optimal pricing in repeated auctions has attracted a large body of research. While dynamic mechanisms offer powerful techniques to improve on both revenue and ef…