Optimal Mixture Weights for Off-Policy Evaluation with Multiple Behavior Policies
Off-policy evaluation is a key component of reinforcement learning which evaluates a target policy with offline data collected from behavior policies. It is a crucial step towards safe reinforcement learning and has been used in advertisement, recommender systems and many other applications. In these applications, sometimes the offline data is collected from multiple behavior policies. Previous works regard data from different behavior policies equally. Nevertheless, some behavior policies are better at producing good estimators while others are not. This paper starts with discussing how to correctly mix estimators produced by different behavior policies. We propose three ways to reduce the variance of the mixture estimator when all sub-estimators are unbiased or asymptotically unbiased. Furthermore, experiments on simulated recommender systems show that our methods are effective in reducing the Mean-Square Error of estimation.
Code (0)
등록된 구현이 없습니다.
Tasks
Off-policy evaluationRecommendation Systemsreinforcement-learningReinforcement LearningReinforcement Learning (RL)Safe Reinforcement LearningSimilar Papers 제목 키워드 기반
Be More Diverse than the Most Diverse: Optimal Mixtures of Generative Models via Mixture-UCB Bandit Algorithms
The availability of multiple training algorithms and architectures for generative models requires a selection mechanism to form a single model over a group of well-trained generation models. The selection task is commonl…
Balancing Immediate Revenue and Future Off-Policy Evaluation in Coupon Allocation
Coupon allocation drives customer purchases and boosts revenue. However, it presents a fundamental trade-off between exploiting the current optimal policy to maximize immediate revenue and exploring alternative policies …
Off-policy evaluationOff-policy Maximum Entropy Reinforcement Learning : Soft Actor-Critic with Advantage Weighted Mixture Policy(SAC-AWMP)
The optimal policy of a reinforcement learning problem is often discontinuous and non-smooth. I.e., for two states with similar representations, their optimal policies can be significantly different. In this case, repres…
continuous-controlContinuous ControlMixture-of-ExpertsReinforcement LearningPreference-centric Bandits: Optimality of Mixtures and Regret-efficient Algorithms
The objective of canonical multi-armed bandits is to identify and repeatedly select an arm with the largest reward, often in the form of the expected value of the arm's probability distribution. Such a utilitarian perspe…
Multi-Armed BanditsNavigateFactor-Aware Mixture-of-Experts with Pretrained Encoder for Combinatorial Generalization
The integration of pretrained encoders with diffusion policies has become a dominant paradigm for visual robotic manipulation. However, it still struggles to generalize across complex environments with varying factors su…