Beyond The Rainbow: High Performance Deep Reinforcement Learning on a Desktop PC

2024-11-06 · Tyler Clark, Mark Towers, Christine Evers, Jonathon Hare

Rainbow Deep Q-Network (DQN) demonstrated combining multiple independent enhancements could significantly boost a reinforcement learning (RL) agent's performance. In this paper, we present "Beyond The Rainbow" (BTR), a novel algorithm that integrates six improvements from across the RL literature to Rainbow DQN, establishing a new state-of-the-art for RL using a desktop PC, with a human-normalized interquartile mean (IQM) of 7.4 on Atari-60. Beyond Atari, we demonstrate BTR's capability to handle complex 3D games, successfully training agents to play Super Mario Galaxy, Mario Kart, and Mortal Kombat with minimal algorithmic changes. Designing BTR with computational efficiency in mind, agents can be trained using a high-end desktop PC on 200 million Atari frames within 12 hours. Additionally, we conduct detailed ablation studies of each component, analyzing the performance and impact using numerous measures. Code is available at https://github.com/VIPTankz/BTR.

📄 PDF Abstract BibTeX arXiv:2411.03820

Code (3)

viptankz/btr 공식 구현 pytorch

rickyegl/BeyondTheRainbowICLR pytorch

rickyegl/BeyondTheRainbowM pytorch

Tasks

Computational EfficiencyDeep Reinforcement LearningReinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Prioritized Experience Replay Prioritized Experience Replay is a type of experience replay in reinforcement learning where we more frequently replay…

Dueling Network A Dueling Network is a type of Q-Network that has two streams to separately estimate (scalar) state-value and the advantages for each action. Both streams share a common…

Convolution A convolution is a type of matrix operation, consisting of a kernel, a small matrix of weights, that slides over input data performing element-wise multiplication with the…

N-step Returns $n$-step Returns are used for value function estimation in reinforcement learning. Specifically, for $n$ steps we can write the complete return as: $$ R\_{t}^{(n)} =…

Noisy Linear Layer A Noisy Linear Layer is a linear layer with parametric noise added to the weights. This induced stochasticity can be used in…

Double Q-learning Double Q-learning is an off-policy reinforcement learning algorithm that utilises double estimation to counteract overestimation problems with traditional Q-learning. The…

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

To Combine or Not To Combine? A Rainbow Deep Reinforcement Learning Agent for Dialog Policies

2019-09-01 · WS 2019 9 · Dirk V{\"a}th, Ngoc Thang Vu

In this paper, we explore state-of-the-art deep reinforcement learning methods for dialog policy training such as prioritized experience replay, double deep Q-Networks, dueling network architectures and distributional le…

Deep Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Is Deep Reinforcement Learning Really Superhuman on Atari? Leveling the playing field

2019-08-13 · Marin Toromanoff, Emilie Wirbel, Fabien Moutarde

Consistent and reproducible evaluation of Deep Reinforcement Learning (DRL) is not straightforward. In the Arcade Learning Environment (ALE), small changes in environment parameters such as stochasticity or the maximum a…

Atari Games Deep Reinforcement Learning General Reinforcement Learning reinforcement-learning+2

Learn to Interpret Atari Agents

2018-12-29 · Zhao Yang, Song Bai, Li Zhang, Philip H. S. Torr

Deep reinforcement learning (DeepRL) agents surpass human-level performance in many tasks. However, the direct mapping from states to actions makes it hard to interpret the rationale behind the decision-making of the age…

Decision Making Deep Reinforcement Learning Reinforcement Learning

Revisiting Rainbow: Promoting more Insightful and Inclusive Deep Reinforcement Learning Research

2020-11-20 · Johan S. Obando-Ceron, Pablo Samuel Castro

Since the introduction of DQN, a vast majority of reinforcement learning research has focused on reinforcement learning with deep neural networks as function approximators. New methods are typically evaluated on a set of…

Atari Games Deep Reinforcement Learning reinforcement-learning Reinforcement Learning+1

Faster Deep Reinforcement Learning with Slower Online Network

2021-12-10 · Kavosh Asadi, Rasool Fakoor, Omer Gottesman, Taesup Kim 외

Deep reinforcement learning algorithms often use two networks for value function optimization: an online network, and a target network that tracks the online network with some delay. Using two separate networks enables t…