Off-Policy Evaluation in Embedded Spaces
Off-policy evaluation methods are important in recommendation systems and search engines, where data collected under an existing logging policy is used to estimate the performance of a new proposed policy. A common approach to this problem is weighting, where data is weighted by a density ratio between the probability of actions given contexts in the target and logged policies. In practice, two issues often arise. First, many problems have very large action spaces and we may not observe rewards for most actions, and so in finite samples we may encounter a positivity violation. Second, many recommendation systems are not probabilistic and so having access to logging and target policy densities may not be feasible. To address these issues, we introduce the featurized embedded permutation weighting estimator. The estimator computes the density ratio in an action embedding space, which reduces the possibility of positivity violations. The density ratio is computed leveraging recent advances in normalizing flows and density ratio estimation as a classification problem, in order to obtain estimates which are feasible in practice.
Code (0)
등록된 구현이 없습니다.
Tasks
Density Ratio EstimationOff-policy evaluationRecommendation SystemsMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Representation of Reinforcement Learning Policies in Reproducing Kernel Hilbert Spaces
We propose a general framework for policy representation for reinforcement learning tasks. This framework involves finding a low-dimensional embedding of the policy on a reproducing kernel Hilbert space (RKHS). The usage…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Jointly-Learned State-Action Embedding for Efficient Reinforcement Learning
While reinforcement learning has achieved considerable successes in recent years, state-of-the-art models are often still limited by the size of state and action spaces. Model-free reinforcement learning approaches use s…
Model-based Reinforcement LearningRecommendation Systemsreinforcement-learningReinforcement Learning+1Jointly-Trained State-Action Embedding for Efficient Reinforcement Learning
While reinforcement learning has achieved considerable successes in recent years, state-of-the-art models are often still limited by the size of state and action spaces. Model-free reinforcement learning approaches use s…
Model-based Reinforcement LearningRecommendation Systemsreinforcement-learningReinforcement Learning+1Comparison of Embedded Spaces for Deep Learning Classification
Embedded spaces are a key feature in deep learning. Good embedded spaces represent the data well to support classification and advanced techniques such as open-set recognition, few-short learning and explainability. This…
ClassificationDeep LearningOpen Set LearningOn the Geometry of Reinforcement Learning in Continuous State and Action Spaces
Advances in reinforcement learning have led to its successful application in complex tasks with continuous state and action spaces. Despite these advances in practice, most theoretical work pertains to finite state and a…
MuJoCoreinforcement-learningReinforcement Learning (RL)