Striving for Simplicity in Off-Policy Deep Reinforcement Learning
This paper advocates the use of offline (batch) reinforcement learning (RL) to help (1) isolate the contributions of exploitation vs. exploration in off-policy deep RL, (2) improve reproducibility of deep RL research, and (3) facilitate the design of simpler deep RL algorithms. We propose an offline RL benchmark on Atari 2600 games comprising all of the replay data of a DQN agent. Using this benchmark, we demonstrate that recent off-policy deep RL algorithms, even when trained solely on logged DQN data, can outperform online DQN. We present Random Ensemble Mixture (REM), a simple Q-learning algorithm that enforces optimal Bellman consistency on random convex combinations of multiple Q-value estimates. The REM algorithm outperforms more complex RL agents such as C51 and QR-DQN on the offline Atari benchmark and performs comparably in the online setting.
Code (0)
등록된 구현이 없습니다.
Tasks
Atari GamesDeep Reinforcement LearningOffline RLQ-Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
DDPG++: Striving for Simplicity in Continuous-control Off-Policy Reinforcement Learning
This paper prescribes a suite of techniques for off-policy Reinforcement Learning (RL) that simplify the training process and reduce the sample complexity. First, we show that simple Deterministic Policy Gradient works r…
continuous-controlContinuous ControlMuJoCoreinforcement-learning+1Striving for Simplicity and Performance in Off-Policy DRL: Output Normalization and Non-Uniform Sampling
We aim to develop off-policy DRL algorithms that not only exceed state-of-the-art performance but are also simple and minimalistic. For standard continuous control benchmarks, Soft Actor-Critic (SAC), which employs entro…
continuous-controlContinuous ControlDRAS-CQSim: A Reinforcement Learning based Framework for HPC Cluster Scheduling
For decades, system administrators have been striving to design and tune cluster scheduling policies to improve the performance of high performance computing (HPC) systems. However, the increasingly complex HPC systems c…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)SchedulingImplicit Policy for Reinforcement Learning
We introduce Implicit Policy, a general class of expressive policies that can flexibly represent complex action distributions in reinforcement learning, with efficient algorithms to compute entropy regularized policy gra…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Cold-Start Reinforcement Learning with Softmax Policy Gradient
Policy-gradient approaches to reinforcement learning have two common and undesirable overhead procedures, namely warm-start training and sample variance reduction. In this paper, we describe a reinforcement learning meth…
Image CaptioningPolicy Gradient Methodsreinforcement-learningReinforcement Learning+1