ReVar: Strengthening Policy Evaluation via Reduced Variance Sampling
This paper studies the problem of data collection for policy evaluation in Markov decision processes (MDPs). In policy evaluation, we are given a target policy and asked to estimate the expected cumulative reward it will obtain in an environment formalized as an MDP. We develop theory for optimal data collection within the class of tree-structured MDPs by first deriving an oracle data collection strategy that uses knowledge of the variance of the reward distributions. We then introduce the Reduced Variance Sampling (ReVar) algorithm that approximates the oracle strategy when the reward variances are unknown a priori and bound its sub-optimality compared to the oracle strategy. Finally, we empirically validate that ReVar leads to policy evaluation with mean squared error comparable to the oracle strategy and significantly lower than simply running the target policy.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
RestoreVAR: Visual Autoregressive Generation for All-in-One Image Restoration
The use of latent diffusion models (LDMs) such as Stable Diffusion has significantly improved the perceptual quality of All-in-One image Restoration (AiOR) methods, while also enhancing their generalization capabilities.…
AllDenoisingImage GenerationImage RestorationPAGE-PG: A Simple and Loopless Variance-Reduced Policy Gradient Method with Probabilistic Gradient Estimation
Despite their success, policy gradient methods suffer from high variance of the gradient estimate, which can result in unsatisfactory sample complexity. Recently, numerous variance-reduced extensions of policy gradient m…
Policy Gradient MethodsVariance-Reduced Off-Policy TDC Learning: Non-Asymptotic Convergence Analysis
Variance reduction techniques have been successfully applied to temporal-difference (TD) learning and help to improve the sample complexity in policy evaluation. However, the existing work applied variance reduction to e…
Data driven synthetic wavefront generation for boundary layer data
Disturbances such as atmospheric turbulence and aero-optic effects lead to wavefront aberrations, which degrade performance in imaging and laser propagation applications. Adaptive optics (AO) provide a method to mitigate…
Time SeriesVariance-Reduced Off-Policy Memory-Efficient Policy Search
Off-policy policy optimization is a challenging problem in reinforcement learning (RL). The algorithms designed for this problem often suffer from high variance in their estimators, which results in poor sample efficienc…
Reinforcement Learning (RL)Stochastic Optimization