Improving Value Estimation Critically Enhances Vanilla Policy Gradient
Modern policy gradient algorithms, such as TRPO and PPO, outperform vanilla policy gradient in many RL tasks. Questioning the common belief that enforcing approximate trust regions leads to steady policy improvement in practice, we show that the more critical factor is the enhanced value estimation accuracy from more value update steps in each iteration. To demonstrate, we show that by simply increasing the number of value update steps per iteration, vanilla policy gradient itself can achieve performance comparable to or better than PPO in all the standard continuous control benchmark environments. Importantly, this simple change to vanilla policy gradient is significantly more robust to hyperparameter choices, opening up the possibility that RL algorithms may still become more effective and easier to use.
Code (1)
Tasks
continuous-controlContinuous ControlMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Offline Minimax Soft-Q-learning Under Realizability and Partial Coverage
In offline reinforcement learning (RL) we have no opportunity to explore so we must make assumptions that the data is sufficient to guide picking a good policy, taking the form of assuming some coverage, realizability, B…
Offline RLQ-LearningReinforcement Learning (RL)Improving Policy Exploitation in Online Reinforcement Learning with Instant Retrospect Action
Existing value-based online reinforcement learning (RL) algorithms suffer from slow policy exploitation due to ineffective exploration and delayed policy updates. To address these challenges, we propose an algorithm call…
Representation LearningReinforcement LearningContinuous ControlCollaborative Weighting with Pessimistic Critic for Mitigating Overestimation in Off-Policy Reinforcement Learning
Deep off-policy reinforcement learning algorithms for continuous control typically rely on neural value function approximation to guide policy improvement. However, temporal-difference (TD) learning introduces noisy targ…
Reinforcement LearningContinuous ControlADER:Adapting between Exploration and Robustness for Actor-Critic Methods
Combining off-policy reinforcement learning methods with function approximators such as neural networks has been found to lead to overestimation of the value function and sub-optimal solutions. Improvement such as TD3 ha…
continuous-controlContinuous ControlImplicit Two-Tower Policies
We present a new class of structured reinforcement learning policy-architectures, Implicit Two-Tower (ITT) policies, where the actions are chosen based on the attention scores of their learnable latent representations wi…
OpenAI GymVocal Bursts Valence Prediction