Reward Scale Robustness for Proximal Policy Optimization via DreamerV3 Tricks

2023-10-26 · NeurIPS 2023 11

Most reinforcement learning methods rely heavily on dense, well-normalized environment rewards. DreamerV3 recently introduced a model-based method with a number of tricks that mitigate these limitations, achieving state-of-the-art on a wide range of benchmarks with a single set of hyperparameters. This result sparked discussion about the generality of the tricks, since they appear to be applicable to other reinforcement learning algorithms. Our work applies DreamerV3's tricks to PPO and is the first such empirical study outside of the original work. Surprisingly, we find that the tricks presented do not transfer as general improvements to PPO. We use a high quality PPO reference implementation and present extensive ablation studies totaling over 10,000 A100 hours on the Arcade Learning Environment and the DeepMind Control Suite. Though our experiments demonstrate that these tricks do not generally outperform PPO, we identify cases where they succeed and offer insight into the relationship between the implementation tricks. In particular, PPO with these tricks performs comparably to PPO on Atari games with reward clipping and significantly outperforms PPO without reward clipping.

📄 PDF Abstract BibTeX arXiv:2310.17805

Code (0)

등록된 구현이 없습니다.

Tasks

Atari Games reinforcement-learning Reinforcement Learning

Methods 이 논문이 사용한 방법론

SET Dynamic Sparse Training method where weight mask is updated randomly periodically

Entropy Regularization 설명 없음

PPO Proximal Policy Optimization, or PPO, is a policy gradient method for reinforcement learning. The motivation was to have an algorithm with the data efficiency and reliable…

Pairwise Proximal Policy Optimization: Harnessing Relative Feedback for LLM Alignment

2023-09-30 · Tianhao Wu, Banghua Zhu, Ruoyu Zhang, Zhaojin Wen 외

Large Language Models (LLMs) can acquire extensive world knowledge through pre-training on large corpora. However, due to exposure to low-quality data, LLMs may exhibit harmful behavior without aligning with human values…

reinforcement-learning Reinforcement Learning World Knowledge

Optimistic Proximal Policy Optimization

2019-06-25 · Takahisa Imagawa, Takuya Hiraoka, Yoshimasa Tsuruoka

Reinforcement Learning, a machine learning framework for training an autonomous agent based on rewards, has shown outstanding results in various domains. However, it is known that learning a good policy is difficult in a…

BIG-bench Machine Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Proximal Policy Optimization via Enhanced Exploration Efficiency

2020-11-11 · Junwei Zhang, Zhenghao Zhang, Shuai Han, Shuai Lü

Proximal policy optimization (PPO) algorithm is a deep reinforcement learning algorithm with outstanding performance, especially in continuous control tasks. But the performance of this method is still affected by its ex…

continuous-control Continuous Control Deep Reinforcement Learning MuJoCo+3

Quantile-Based Policy Optimization for Reinforcement Learning

2022-01-27 · Jinyang Jiang, Jiaqiao Hu, Yijie Peng

Classical reinforcement learning (RL) aims to optimize the expected cumulative rewards. In this work, we consider the RL setting where the goal is to optimize the quantile of the cumulative rewards. We parameterize the p…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Proximal Policy Optimization with Mixed Distributed Training

2019-07-15 · Zhen-Yu Zhang, Xiangfeng Luo, Tong Liu, Shaorong Xie 외

Instability and slowness are two main problems in deep reinforcement learning. Even if proximal policy optimization (PPO) is the state of the art, it still suffers from these two problems. We introduce an improved algori…

Deep Reinforcement Learning Reinforcement Learning