Proximal Policy Optimization Algorithms
We propose a new family of policy gradient methods for reinforcement learning, which alternate between sampling data through interaction with the environment, and optimizing a "surrogate" objective function using stochastic gradient ascent. Whereas standard policy gradient methods perform one gradient update per data sample, we propose a novel objective function that enables multiple epochs of minibatch updates. The new methods, which we call proximal policy optimization (PPO), have some of the benefits of trust region policy optimization (TRPO), but they are much simpler to implement, more general, and have better sample complexity (empirically). Our experiments test PPO on a collection of benchmark tasks, including simulated robotic locomotion and Atari game playing, and we show that PPO outperforms other online policy gradient methods, and overall strikes a favorable balance between sample complexity, simplicity, and wall-time.
Code (188)
Tasks
Continuous ControlDota 2Multi-Goal Reinforcement LearningNeural Architecture SearchOpenAI GymPolicy Gradient MethodsReinforcement LearningSpeech EnhancementMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Hindsight Experience Replay Accelerates Proximal Policy Optimization
Hindsight experience replay (HER) accelerates off-policy reinforcement learning algorithms for environments that emit sparse rewards by modifying the goal of the episode post-hoc to be some state achieved during the epis…
reinforcement-learningReinforcement LearningGeneralized Proximal Policy Optimization with Sample Reuse
In real-world decision making tasks, it is critical for data-driven reinforcement learning methods to be both stable and sample efficient. On-policy methods typically generate reliable policy improvement throughout train…
Decision MakingProximal Deterministic Policy Gradient
This paper introduces two simple techniques to improve off-policy Reinforcement Learning (RL) algorithms. First, we formulate off-policy RL as a stochastic proximal point iteration. The target network plays the role of t…
continuous-controlContinuous Controlreinforcement-learningReinforcement Learning (RL)A-3PO: Accelerating Asynchronous LLM Training with Staleness-aware Proximal Policy Approximation
Decoupled PPO has been a successful reinforcement learning (RL) algorithm to deal with the high data staleness under the asynchronous RL setting. Decoupled loss used in decoupled PPO improves coupled-loss style of algori…
Reinforcement LearningBehavior Proximal Policy Optimization
Offline reinforcement learning (RL) is a challenging setting where existing off-policy actor-critic methods perform poorly due to the overestimation of out-of-distribution state-action pairs. Thus, various additional aug…
D4RLOffline RLReinforcement Learning (RL)