Efficient Parallel Reinforcement Learning Framework using the Reactor Model

2023-12-07 · Jacky Kwok, Marten Lohstroh, Edward A. Lee

Parallel Reinforcement Learning (RL) frameworks are essential for mapping RL workloads to multiple computational resources, allowing for faster generation of samples, estimation of values, and policy improvement. These computational paradigms require a seamless integration of training, serving, and simulation workloads. Existing frameworks, such as Ray, are not managing this orchestration efficiently, especially in RL tasks that demand intensive input/output and synchronization between actors on a single node. In this study, we have proposed a solution implementing the reactor model, which enforces a set of actors to have a fixed communication pattern. This allows the scheduler to eliminate work needed for synchronization, such as acquiring and releasing locks for each actor or sending and processing coordination-related messages. Our framework, Lingua Franca (LF), a coordination language based on the reactor model, also supports true parallelism in Python and provides a unified interface that allows users to automatically generate dataflow graphs for RL tasks. In comparison to Ray on a single-node multi-core compute platform, LF achieves 1.21x and 11.62x higher simulation throughput in OpenAI Gym and Atari environments, reduces the average training time of synchronized parallel Q-learning by 31.2%, and accelerates multi-agent RL inference by 5.12x.

📄 PDF Abstract BibTeX arXiv:2312.04704

Code (1)

jackyk02/distributed_rl_benchmarks 공식 구현

Tasks

OpenAI GymQ-Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

SET Dynamic Sparse Training method where weight mask is updated randomly periodically

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

Design Optimization of Nuclear Fusion Reactor through Deep Reinforcement Learning

2024-09-12 · Jinsu Kim, Jaemin Seo

This research explores the application of Deep Reinforcement Learning (DRL) to optimize the design of a nuclear fusion reactor. DRL can efficiently address the challenging issues attributed to multiple physics and engine…

Deep Reinforcement Learning reinforcement-learning Reinforcement Learning

Reactor Optimization Benchmark by Reinforcement Learning

2024-03-21 · Deborah Schwarcz, Nadav Schneider, Gal Oren, Uri Steinitz

Neutronic calculations for reactors are a daunting task when using Monte Carlo (MC) methods. As high-performance computing has advanced, the simulation of a reactor is nowadays more readily done, but design and optimizat…

reinforcement-learning Reinforcement Learning

Model predictive control and moving horizon estimation for adaptive optimal bolus feeding in high-throughput cultivation of \textit{E. coli}

2022-03-14 · Jong Woo Kim, Niels Krausch, Judit Aizpuru, Tilman Barz 외

We discuss the application of a nonlinear model predictive control (MPC) and a moving horizon estimation (MHE) to achieve an optimal operation of \textit{E. coli} fed-batch cultivations with intermittent bolus feeding. 2…

Model Predictive Control

Multi-Mode Process Control Using Multi-Task Inverse Reinforcement Learning

2025-05-27 · Runze Lin, Junghui Chen, Biao Huang, Lei Xie 외

In the era of Industry 4.0 and smart manufacturing, process systems engineering must adapt to digital transformation. While reinforcement learning offers a model-free approach to process control, its applications are lim…

Multi-Task Learning reinforcement-learning Reinforcement Learning

Trustworthy AI for Process Automation on a Chylla-Haase Polymerization Reactor

2021-08-30 · Daniel Hein, Daniel Labisch

In this paper, genetic programming reinforcement learning (GPRL) is utilized to generate human-interpretable control policies for a Chylla-Haase polymerization reactor. Such continuously stirred tank reactors (CSTRs) wit…