Efficient Baseline-free Sampling in Parameter Exploring Policy Gradients: Super Symmetric PGPE
Policy Gradient methods that explore directly in parameter space are among the most effective and robust direct policy search methods and have drawn a lot of attention lately. The basic method from this field, Policy Gradients with Parameter-based Exploration, uses two samples that are symmetric around the current hypothesis to circumvent misleading reward in \emph{asymmetrical} reward distributed problems gathered with the usual baseline approach. The exploration parameters are still updated by a baseline approach - leaving the exploration prone to asymmetric reward distributions. In this paper we will show how the exploration parameters can be sampled quasi symmetric despite having limited instead of free parameters for exploration. We give a transformation approximation to get quasi symmetric samples with respect to the exploration without changing the overall sampling distribution. Finally we will demonstrate that sampling symmetrically also for the exploration parameters is superior in needs of samples and robustness than the original sampling approach.
Code (0)
등록된 구현이 없습니다.
Tasks
Policy Gradient MethodsSimilar Papers 제목 키워드 기반
Conservative Optimistic Policy Optimization via Multiple Importance Sampling
Reinforcement Learning (RL) has been able to solve hard problems such as playing Atari games or solving the game of Go, with a unified approach. Yet modern deep RL approaches are still not widely used in real-world appli…
Atari GamesGame of Goreinforcement-learningReinforcement Learning (RL)Improving Policy Gradient by Exploring Under-appreciated Rewards
This paper presents a novel form of policy gradient for model-free reinforcement learning (RL) with improved exploration properties. Current policy-based methods use entropy regularization to encourage undirected explora…
Reinforcement LearningReinforcement Learning (RL)Langevin Soft Actor-Critic: Efficient Exploration through Uncertainty-Driven Critic Learning
Existing actor-critic algorithms, which are popular for continuous control reinforcement learning (RL) tasks, suffer from poor sample efficiency due to lack of principled exploration mechanism within them. Motivated by t…
continuous-controlContinuous ControlEfficient ExplorationReinforcement Learning (RL)+1Importance Resampling for Off-policy Policy Evaluation
Importance sampling is a common approach to off-policy learning in reinforcement learning. While it is consistent and unbiased, it can result in high variance updates to the parameters for the value function. Weighted i…
Variance Reduced Domain Randomization for Policy Gradient
By introducing randomness on environment parameters that fundamentally affect the dynamics, domain randomization (DR) imposes diversity to the policy trained by deep reinforcement learning, and thus improves its capabili…
Deep Reinforcement LearningPolicy Gradient Methods