Importance Resampling for Off-policy Prediction
Importance sampling (IS) is a common reweighting strategy for off-policy prediction in reinforcement learning. While it is consistent and unbiased, it can result in high variance updates to the weights for the value function. In this work, we explore a resampling strategy as an alternative to reweighting. We propose Importance Resampling (IR) for off-policy prediction, which resamples experience from a replay buffer and applies standard on-policy updates. The approach avoids using importance sampling ratios in the update, instead correcting the distribution before the update. We characterize the bias and consistency of IR, particularly compared to Weighted IS (WIS). We demonstrate in several microworlds that IR has improved sample efficiency and lower variance updates, as compared to IS and several variance-reduced IS strategies, including variants of WIS and V-trace which clips IS ratios. We also provide a demonstration showing IR improves over IS for learning a value function from images in a racing car simulator.
Code (2)
Tasks
PredictionReinforcement LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Importance Resampling for Off-policy Policy Evaluation
Importance sampling is a common approach to off-policy learning in reinforcement learning. While it is consistent and unbiased, it can result in high variance updates to the parameters for the value function. Weighted i…
Observation Adaptation via Annealed Importance Resampling for Partially Observable Markov Decision Processes
Partially observable Markov decision processes (POMDPs) are a general mathematical model for sequential decision-making in stochastic environments under state uncertainty. POMDPs are often solved \textit{online}, which e…
Decision MakingSequential Decision MakingEvaluating Control Protocols for Untrusted AI Agents
As AI systems become more capable and widely deployed as agents, ensuring their safe operation becomes critical. AI control offers one approach to mitigating the risk from untrusted AI agents by monitoring their actions …
Kernel Metric Learning for In-Sample Off-Policy Evaluation of Deterministic RL Policies
We consider off-policy evaluation (OPE) of deterministic target policies for reinforcement learning (RL) in environments with continuous action spaces. While it is common to use importance sampling for OPE, it suffers fr…
Metric LearningOff-policy evaluationReinforcement Learning (RL)Importance weighting without importance weights: An efficient algorithm for combinatorial semi-bandits
We propose a sample-efficient alternative for importance weighting for situations where one only has sample access to the probability distribution that generates the observations. Our new method, called Geometric Resampl…
Combinatorial Optimization