Learning from Logged Implicit Exploration Data
We provide a sound and consistent foundation for the use of \emph{nonrandom} exploration data in "contextual bandit" or "partially labeled" settings where only the value of a chosen action is learned. The primary challenge in a variety of settings is that the exploration policy, in which "offline" data is logged, is not explicitly known. Prior solutions here require either control of the actions during the learning process, recorded random exploration, or actions chosen obliviously in a repeated manner. The techniques reported here lift these restrictions, allowing the learning of a policy for choosing actions given features from historical data where no randomization occurred or was logged. We empirically verify our solution on two reasonably sized sets of real-world data obtained from Yahoo!.
Code (1)
Similar Papers 제목 키워드 기반
Top-K Off-Policy Correction for a REINFORCE Recommender System
Industrial recommender systems deal with extremely large action spaces -- many millions of items to recommend. Moreover, they need to serve billions of users, who are unique at any point in time, making a complex user st…
Recommendation SystemsCombining Online Learning and Offline Learning for Contextual Bandits with Deficient Support
We address policy learning with logged data in contextual bandits. Current offline-policy learning algorithms are mostly based on inverse propensity score (IPS) weighting requiring the logging policy to have \emph{full s…
Multi-Armed BanditsStriving for Simplicity in Off-Policy Deep Reinforcement Learning
This paper advocates the use of offline (batch) reinforcement learning (RL) to help (1) isolate the contributions of exploitation vs. exploration in off-policy deep RL, (2) improve reproducibility of deep RL research, an…
Atari GamesDeep Reinforcement LearningOffline RLQ-Learning+3A General Offline Reinforcement Learning Framework for Interactive Recommendation
This paper studies the problem of learning interactive recommender systems from logged feedbacks without any exploration in online environments. We address the problem by proposing a general offline reinforcement learnin…
Interactive RecommendationRecommendation Systemsreinforcement-learningReinforcement LearningActive Learning with Logged Data
We consider active learning with logged data, where labeled examples are drawn conditioned on a predetermined logging policy, and the goal is to learn a classifier on the entire population, not just conditioned on the lo…
Active Learning