The Reactor: A fast and sample-efficient Actor-Critic agent for Reinforcement Learning
In this work we present a new agent architecture, called Reactor, which combines multiple algorithmic and architectural contributions to produce an agent with higher sample-efficiency than Prioritized Dueling DQN (Wang et al., 2016) and Categorical DQN (Bellemare et al., 2017), while giving better run-time performance than A3C (Mnih et al., 2016). Our first contribution is a new policy evaluation algorithm called Distributional Retrace, which brings multi-step off-policy updates to the distributional reinforcement learning setting. The same approach can be used to convert several classes of multi-step policy evaluation algorithms designed for expected value evaluation into distributional ones. Next, we introduce the \b{eta}-leave-one-out policy gradient algorithm which improves the trade-off between variance and bias by using action values as a baseline. Our final algorithmic contribution is a new prioritized replay algorithm for sequences, which exploits the temporal locality of neighboring observations for more efficient replay prioritization. Using the Atari 2600 benchmarks, we show that each of these innovations contribute to both the sample efficiency and final agent performance. Finally, we demonstrate that Reactor reaches state-of-the-art performance after 200 million frames and less than a day of training.
Code (0)
등록된 구현이 없습니다.
Tasks
Atari GamesDistributional Reinforcement LearningReinforcement LearningReinforcement Learning (RL)Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Multistep Criticality Search and Power Shaping in Microreactors with Reinforcement Learning
Reducing operation and maintenance costs is a key objective for advanced reactors in general and microreactors in particular. To achieve this reduction, developing robust autonomous control algorithms is essential to ens…
energy managementReinforcement Learning (RL)Deep Surrogate Models for Multi-dimensional Regression of Reactor Power
There is renewed interest in developing small modular reactors and micro-reactors. Innovation is necessary in both construction and operation methods of these reactors to be financially attractive. For operation, an area…
regressionEfficient Parallel Reinforcement Learning Framework using the Reactor Model
Parallel Reinforcement Learning (RL) frameworks are essential for mapping RL workloads to multiple computational resources, allowing for faster generation of samples, estimation of values, and policy improvement. These c…
OpenAI GymQ-Learningreinforcement-learningReinforcement Learning+1Assessing the viability of non-light water reactor concepts for electricity and heat generation in decarbonized energy systems
Recent pledges to triple global nuclear capacity by 2050 suggest a "nuclear renaissance," bolstered by reactor concepts such as sodium-cooled fast reactors, high-temperature reactors, and molten salt reactors. These tech…
Digital Twin-Centered Hybrid Data-Driven Multi-Stage Deep Learning Framework for Enhanced Nuclear Reactor Power Prediction
The accurate and efficient modeling of nuclear reactor transients is crucial for ensuring safe and optimal reactor operation. Traditional physics-based models, while valuable, can be computationally intensive and may not…
Decision Making