Fast Population-Based Reinforcement Learning on a Single Machine

2022-06-17 · Arthur Flajolet, Claire Bizon Monroc, Karim Beguir, Thomas Pierrot

Training populations of agents has demonstrated great promise in Reinforcement Learning for stabilizing training, improving exploration and asymptotic performance, and generating a diverse set of solutions. However, population-based training is often not considered by practitioners as it is perceived to be either prohibitively slow (when implemented sequentially), or computationally expensive (if agents are trained in parallel on independent accelerators). In this work, we compare implementations and revisit previous studies to show that the judicious use of compilation and vectorization allows population-based training to be performed on a single machine with one accelerator with minimal overhead compared to training a single agent. We also show that, when provided with a few accelerators, our protocols extend to large population sizes for applications such as hyperparameter tuning. We hope that this work and the public release of our code will encourage practitioners to use population-based learning more frequently for their research and applications.

📄 PDF Abstract BibTeX arXiv:2206.08888

Code (1)

instadeepai/fastpbrl jax

Tasks

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Population Based Training of Neural Networks

2017-11-27 · Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M. Czarnecki 외

Neural networks dominate the modern machine learning landscape, but their training and success still suffer from sensitivity to empirical choices of hyperparameters such as model architecture, loss function, and optimisa…

Deep Reinforcement Learning Machine Translation Model Selection Reinforcement Learning

Simultaneous Training of First- and Second-Order Optimizers in Population-Based Reinforcement Learning

2024-08-27 · Felix Pfeiffer, Shahram Eivazi

The tuning of hyperparameters in reinforcement learning (RL) is critical, as these parameters significantly impact an agent's performance and learning efficiency. Dynamic adjustment of hyperparameters during the training…

MuJoCo Reinforcement Learning (RL)

Learning Meta Representations for Agents in Multi-Agent Reinforcement Learning

2021-08-30 · Shenao Zhang, Lei Han, Li Shen

In multi-agent reinforcement learning, the behaviors that agents learn in a single Markov Game (MG) are typically confined to the given agent number. Every single MG induced by varying the population may possess distinct…

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Recruitment-imitation Mechanism for Evolutionary Reinforcement Learning

2019-12-13 · Shuai Lü, Shuai Han, Wenbo Zhou, Junwei Zhang

Reinforcement learning, evolutionary algorithms and imitation learning are three principal methods to deal with continuous control tasks. Reinforcement learning is sample efficient, yet sensitive to hyper-parameters sett…

continuous-control Continuous Control Efficient Exploration Evolutionary Algorithms+5

ULTHO: Ultra-Lightweight yet Efficient Hyperparameter Optimization in Deep Reinforcement Learning

2025-03-08 · Mingqi Yuan, Bo Li, Xin Jin, Wenjun Zeng

Hyperparameter optimization (HPO) is a billion-dollar problem in machine learning, which significantly impacts the training efficiency and model performance. However, achieving efficient and robust HPO in deep reinforcem…

Bayesian Optimization Deep Reinforcement Learning Hyperparameter Optimization Reinforcement Learning (RL)