Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO
We study the roots of algorithmic progress in deep policy gradient algorithms through a case study on two popular algorithms: Proximal Policy Optimization (PPO) and Trust Region Policy Optimization (TRPO). Specifically, we investigate the consequences of "code-level optimizations:" algorithm augmentations found only in implementations or described as auxiliary details to the core algorithm. Seemingly of secondary importance, such optimizations turn out to have a major impact on agent behavior. Our results show that they (a) are responsible for most of PPO's gain in cumulative reward over TRPO, and (b) fundamentally change how RL methods function. These insights show the difficulty and importance of attributing performance gains in deep reinforcement learning. Code for reproducing our results is available at https://github.com/MadryLab/implementation-matters .
Code (3)
Tasks
Deep Reinforcement Learningreinforcement-learningReinforcement Learning (RL)Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Implementation Matters in Deep RL: A Case Study on PPO and TRPO
We study the roots of algorithmic progress in deep policy gradient algorithms through a case study on two popular algorithms, Proximal Policy Optimization and Trust Region Policy Optimization. We investigate the conseque…
Deep Reinforcement Learningreinforcement-learningReinforcement Learning (RL)Towards Understanding Deep Policy Gradients: A Case Study on PPO
Deep reinforcement learning has shown impressive performance on many decision-making problems, where deep policy gradient algorithms prevail in continuous action space tasks. Although many algorithm-level improvements on…
Decision MakingDeep Reinforcement Learningreinforcement-learningReinforcement Learning (RL)What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study
In recent years, on-policy reinforcement learning (RL) has been successfully applied to many different continuous control tasks. While RL algorithms are often conceptually simple, their state-of-the-art implementations t…
Attributecontinuous-controlContinuous Controlreinforcement-learning+1What Matters for On-Policy Deep Actor-Critic Methods? A Large-Scale Study
In recent years, reinforcement learning (RL) has been successfully applied to many different continuous control tasks. While RL algorithms are often conceptually simple, their state-of-the-art implementations take numero…
Attributecontinuous-controlContinuous ControlReinforcement Learning (RL)Unbiased Risk Estimators Can Mislead: A Case Study of Learning with Complementary Labels
In weakly supervised learning, unbiased risk estimator(URE) is a powerful tool for training classifiers when training and test data are drawn from different distributions. Nevertheless, UREs lead to overfitting in many p…
Weakly-supervised Learning