Sample Reuse via Importance Sampling in Information Geometric Optimization
In this paper we propose a technique to reduce the number of function evaluations, which is often the bottleneck of the black-box optimization, in the information geometric optimization (IGO) that is a generic framework of the probability model-based black-box optimization algorithms and generalizes several well-known evolutionary algorithms, such as the population-based incremental learning (PBIL) and the pure rank-$\mu$ update covariance matrix adaptation evolution strategy (CMA-ES). In each iteration, the IGO algorithms update the parameters of the probability distribution to the natural gradient direction estimated by Monte-Carlo with the samples drawn from the current distribution. Our strategy is to reuse previously generated and evaluated samples based on the importance sampling. It is a technique to reduce the estimation variance without introducing a bias in Monte-Carlo estimation. We apply the sample reuse technique to the PBIL and the pure rank-$\mu$ update CMA-ES and empirically investigate its effect. The experimental results show that the sample reuse helps to reduce the number of function evaluations on many benchmark functions for both the PBIL and the pure rank-$\mu$ update CMA-ES. Moreover, we demonstrate how to combine the importance sampling technique with a variant of the CMA-ES involving an algorithmic component that is not derived in the IGO framework.
Code (0)
등록된 구현이 없습니다.
Tasks
Evolutionary AlgorithmsIncremental LearningSimilar Papers 제목 키워드 기반
On the Reuse Bias in Off-Policy Reinforcement Learning
Importance sampling (IS) is a popular technique in off-policy evaluation, which re-weights the return of trajectories in the replay buffer to boost sample efficiency. However, training with IS can be unstable and previou…
continuous-controlContinuous ControlMuJoCoOff-policy evaluation+3Bayesian experimental design: grouped geometric pooled posterior via ensemble Kalman methods
Bayesian experimental design (BED) for complex physical systems is often limited by the nested inference required to estimate the expected information gain (EIG) or its gradients. Each outer sample induces a different po…
Sample Dropout: A Simple yet Effective Variance Reduction Technique in Deep Policy Optimization
Recent success in Deep Reinforcement Learning (DRL) methods has shown that policy optimization with respect to an off-policy distribution via importance sampling is effective for sample reuse. In this paper, we show that…
Deep Reinforcement LearningMuJoCoOcclusion-Point Reuse for Ray-Traced Ambient Occlusion and Shadow
Ambient occlusion (AO) and soft shadows are critical visibility cues for spatial perception in real-time rendering. Hardware ray tracing provides a direct way to evaluate these effects, enabling ray-traced AO and area-li…
Dimension-Wise Importance Sampling Weight Clipping for Sample-Efficient Reinforcement Learning
In importance sampling (IS)-based reinforcement learning algorithms such as Proximal Policy Optimization (PPO), IS weights are typically clipped to avoid large variance in learning. However, policy update from clipped st…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)