Proximal Policy Optimization via Enhanced Exploration Efficiency

2020-11-11 · Junwei Zhang, Zhenghao Zhang, Shuai Han, Shuai Lü

Proximal policy optimization (PPO) algorithm is a deep reinforcement learning algorithm with outstanding performance, especially in continuous control tasks. But the performance of this method is still affected by its exploration ability. For classical reinforcement learning, there are some schemes that make exploration more full and balanced with data exploitation, but they can't be applied in complex environments due to the complexity of algorithm. Based on continuous control tasks with dense reward, this paper analyzes the assumption of the original Gaussian action exploration mechanism in PPO algorithm, and clarifies the influence of exploration ability on performance. Afterward, aiming at the problem of exploration, an exploration enhancement mechanism based on uncertainty estimation is designed in this paper. Then, we apply exploration enhancement theory to PPO algorithm and propose the proximal policy optimization algorithm with intrinsic exploration module (IEM-PPO) which can be used in complex environments. In the experimental parts, we evaluate our method on multiple tasks of MuJoCo physical simulator, and compare IEM-PPO algorithm with curiosity driven exploration algorithm (ICM-PPO) and original algorithm (PPO). The experimental results demonstrate that IEM-PPO algorithm needs longer training time, but performs better in terms of sample efficiency and cumulative reward, and has stability and robustness.

📄 PDF Abstract BibTeX arXiv:2011.05525

Code (0)

등록된 구현이 없습니다.

Tasks

continuous-control Continuous ControlDeep Reinforcement LearningMuJoCoreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Entropy Regularization 설명 없음

PPO Proximal Policy Optimization, or PPO, is a policy gradient method for reinforcement learning. The motivation was to have an algorithm with the data efficiency and reliable…

Proximal Policy Optimization with Adaptive Exploration

2024-05-07 · Andrei Lixandru

Proximal Policy Optimization with Adaptive Exploration (axPPO) is introduced as a novel learning algorithm. This paper investigates the exploration-exploitation tradeoff within the context of reinforcement learning and a…

reinforcement-learning Reinforcement Learning

Proximal Policy Optimization with Evolutionary Mutations

2026-01-21 · Casimir Czworkowski, Stephen Hornish, Alhassan S. Yasin arxiv

Proximal Policy Optimization (PPO) is a widely used reinforcement learning algorithm known for its stability and sample efficiency, but it often suffers from premature convergence due to limited exploration. In this pape…

Reinforcement Learning OpenAI Gym

Deep Gaussian Process Proximal Policy Optimization

2025-11-22 · Matthijs van der Lende, Juan Cardenas-Cartagena arxiv

Uncertainty estimation for Reinforcement Learning (RL) is a critical component in control tasks where agents must balance safe exploration and efficient learning. While deep neural networks have enabled breakthroughs in …

Reinforcement Learning Continuous Control Gaussian Processes

Multi-Path Policy Optimization

2019-11-11 · Ling Pan, Qingpeng Cai, Longbo Huang

Recent years have witnessed a tremendous improvement of deep reinforcement learning. However, a challenging problem is that an agent may suffer from inefficient exploration, particularly for on-policy methods. Previous e…

Deep Reinforcement Learning Efficient Exploration MuJoCo Reinforcement Learning

Truly Proximal Policy Optimization

2019-03-19 · Yuhui Wang, Hao He, Chao Wen, Xiaoyang Tan

Proximal policy optimization (PPO) is one of the most successful deep reinforcement-learning methods, achieving state-of-the-art performance across a wide range of challenging tasks. However, its optimization behavior is…

Deep Reinforcement Learning Reinforcement Learning