An Efficient Asynchronous Method for Integrating Evolutionary and Gradient-based Policy Search

2020-12-10 · NeurIPS 2020 12 · Kyunghyun Lee, Byeong-Uk Lee, Ukcheol Shin, In So Kweon

Deep reinforcement learning (DRL) algorithms and evolution strategies (ES) have been applied to various tasks, showing excellent performances. These have the opposite properties, with DRL having good sample efficiency and poor stability, while ES being vice versa. Recently, there have been attempts to combine these algorithms, but these methods fully rely on synchronous update scheme, making it not ideal to maximize the benefits of the parallelism in ES. To solve this challenge, asynchronous update scheme was introduced, which is capable of good time-efficiency and diverse policy exploration. In this paper, we introduce an Asynchronous Evolution Strategy-Reinforcement Learning (AES-RL) that maximizes the parallel efficiency of ES and integrates it with policy gradient methods. Specifically, we propose 1) a novel framework to merge ES and DRL asynchronously and 2) various asynchronous update methods that can take all advantages of asynchronism, ES, and DRL, which are exploration and time efficiency, stability, and sample efficiency, respectively. The proposed framework and update methods are evaluated in continuous control benchmark work, showing superior performance as well as time efficiency compared to the previous methods.

📄 PDF Abstract BibTeX arXiv:2012.05417

Code (1)

KyunghyunLee/aes-rl 공식 구현 pytorch

Tasks

continuous-control Continuous ControlDeep Reinforcement LearningPolicy Gradient Methodsreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Lamarckian Platform: Pushing the Boundaries of Evolutionary Reinforcement Learning towards Asynchronous Commercial Games

2022-09-21 · Hui Bai, Ruimin Shen, Yue Lin, Botian Xu 외

Despite the emerging progress of integrating evolutionary computation into reinforcement learning, the absence of a high-performance platform endowing composability and massive parallelism causes non-trivial difficulties…

CPU Distributed Computing reinforcement-learning Reinforcement Learning+2

MAP-Elites with Descriptor-Conditioned Gradients and Archive Distillation into a Single Policy

2023-03-07 · Maxence Faldor, Félix Chalumeau, Manon Flageat, Antoine Cully

Quality-Diversity algorithms, such as MAP-Elites, are a branch of Evolutionary Computation generating collections of diverse and high-performing solutions, that have been successfully applied to a variety of domains and …

Deep Reinforcement Learning

ERL-Re$^2$: Efficient Evolutionary Reinforcement Learning with Shared State Representation and Individual Policy Representation

2022-10-26 · Jianye Hao, Pengyi Li, Hongyao Tang, Yan Zheng 외

Deep Reinforcement Learning (Deep RL) and Evolutionary Algorithms (EA) are two major paradigms of policy optimization with distinct learning principles, i.e., gradient-based v.s. gradient-free. An appealing research dire…

continuous-control Continuous Control Deep Reinforcement Learning Evolutionary Algorithms+2

Proximal Policy Optimization with Evolutionary Mutations

2026-01-21 · Casimir Czworkowski, Stephen Hornish, Alhassan S. Yasin arxiv

Proximal Policy Optimization (PPO) is a widely used reinforcement learning algorithm known for its stability and sample efficiency, but it often suffers from premature convergence due to limited exploration. In this pape…

Reinforcement Learning OpenAI Gym

Evolutionary Policy Optimization

2025-04-17 · Zelal Su "Lain" Mustafaoglu, Keshav Pingali, Risto Miikkulainen

A key challenge in reinforcement learning (RL) is managing the exploration-exploitation trade-off without sacrificing sample efficiency. Policy gradient (PG) methods excel in exploitation through fine-grained, gradient-b…

Policy Gradient Methods Reinforcement Learning (RL)