Accelerating Offline Reinforcement Learning Application in Real-Time Bidding and Recommendation: Potential Use of Simulation
In recommender systems (RecSys) and real-time bidding (RTB) for online advertisements, we often try to optimize sequential decision making using bandit and reinforcement learning (RL) techniques. In these applications, offline reinforcement learning (offline RL) and off-policy evaluation (OPE) are beneficial because they enable safe policy optimization using only logged data without any risky online interaction. In this position paper, we explore the potential of using simulation to accelerate practical research of offline RL and OPE, particularly in RecSys and RTB. Specifically, we discuss how simulation can help us conduct empirical research of offline RL and OPE. We take a position to argue that we should effectively use simulations in the empirical research of offline RL and OPE. To refute the counterclaim that experiments using only real-world data are preferable, we first point out the underlying risks and reproducibility issue in real-world experiments. Then, we describe how these issues can be addressed by using simulations. Moreover, we show how to incorporate the benefits of both real-world and simulation-based experiments to defend our position. Finally, we also present an open challenge to further facilitate practical research of offline RL and OPE in RecSys and RTB, with respect to public simulation platforms. As a possible solution for the issue, we show our ongoing open source project and its potential use case. We believe that building and utilizing simulation-based evaluation platforms for offline RL and OPE will be of great interest and relevance for the RecSys and RTB community.
Code (0)
등록된 구현이 없습니다.
Tasks
Decision MakingOffline RLOff-policy evaluationPositionRecommendation Systemsreinforcement-learningReinforcement Learning (RL)Sequential Decision MakingSimilar Papers 제목 키워드 기반
AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
Reinforcement learning (RL) provides an appealing formalism for learning control policies from experience. However, the classic active formulation of RL necessitates a lengthy active exploration process for each behavior…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Accelerating Diffusion Models in Offline RL via Reward-Aware Consistency Trajectory Distillation
Although diffusion models have achieved strong results in decision-making tasks, their slow inference speed remains a key limitation. While the consistency model offers a potential solution, its applications to decision-…
Decision MakingMuJoCoOffline RLOPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning
Reinforcement learning (RL) has achieved impressive performance in a variety of online settings in which an agent's ability to query the environment for transitions and rewards is effectively unlimited. However, in many …
Few-Shot Imitation LearningImitation LearningOffline RLreinforcement-learning+2Safe Offline Reinforcement Learning with Real-Time Budget Constraints
Aiming at promoting the safe real-world deployment of Reinforcement Learning (RL), research on safe RL has made significant progress in recent years. However, most existing works in the literature still focus on the onli…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Marvel: Accelerating Safe Online Reinforcement Learning with Finetuned Offline Policy
The high costs and risks involved in extensive environment interactions hinder the practical application of current online safe reinforcement learning (RL) methods. While offline safe RL addresses this by learning polici…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Safe Reinforcement Learning