Striving for Simplicity and Performance in Off-Policy DRL: Output Normalization and Non-Uniform Sampling
We aim to develop off-policy DRL algorithms that not only exceed state-of-the-art performance but are also simple and minimalistic. For standard continuous control benchmarks, Soft Actor-Critic (SAC), which employs entropy maximization, currently provides state-of-the-art performance. We first demonstrate that the entropy term in SAC addresses action saturation due to the bounded nature of the action spaces, with this insight, we propose a streamlined algorithm with a simple normalization scheme or with inverted gradients. We show that both approaches can match SAC's sample efficiency performance without the need of entropy maximization, we then propose a simple non-uniform sampling method for selecting transitions from the replay buffer during training. Extensive experimental results demonstrate that our proposed sampling scheme leads to state of the art sample efficiency on challenging continuous control tasks. We combine all of our findings into one simple algorithm, which we call Streamlined Off Policy with Emphasizing Recent Experience, for which we provide robust public-domain code.
Code (3)
Tasks
continuous-controlContinuous ControlMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
DDPG++: Striving for Simplicity in Continuous-control Off-Policy Reinforcement Learning
This paper prescribes a suite of techniques for off-policy Reinforcement Learning (RL) that simplify the training process and reduce the sample complexity. First, we show that simple Deterministic Policy Gradient works r…
continuous-controlContinuous ControlMuJoCoreinforcement-learning+1SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learning
Recent advances in CV and NLP have been largely driven by scaling up the number of network parameters, despite traditional theories suggesting that larger networks are prone to overfitting. These large networks avoid ove…
Computational EfficiencyDeep Reinforcement LearningStriving for Simplicity in Off-Policy Deep Reinforcement Learning
This paper advocates the use of offline (batch) reinforcement learning (RL) to help (1) isolate the contributions of exploitation vs. exploration in off-policy deep RL, (2) improve reproducibility of deep RL research, an…
Atari GamesDeep Reinforcement LearningOffline RLQ-Learning+3Learning Nonparametric High-Dimensional Generative Models: The Empirical-Beta-Copula Autoencoder
By sampling from the latent space of an autoencoder and decoding the latent space samples to the original data space, any autoencoder can simply be turned into a generative model. For this to work, it is necessary to mod…
Batch Normalization with Enhanced Linear Transformation
Batch normalization (BN) is a fundamental unit in modern deep networks, in which a linear transformation module was designed for improving BN's flexibility of fitting complex data distributions. In this paper, we demonst…