Learning Synthetic Environments for Reinforcement Learning with Evolution Strategies

2021-01-24 · Fabio Ferreira, Thomas Nierhoff, Frank Hutter

This work explores learning agent-agnostic synthetic environments (SEs) for Reinforcement Learning. SEs act as a proxy for target environments and allow agents to be trained more efficiently than when directly trained on the target environment. We formulate this as a bi-level optimization problem and represent an SE as a neural network. By using Natural Evolution Strategies and a population of SE parameter vectors, we train agents in the inner loop on evolving SEs while in the outer loop we use the performance on the target task as a score for meta-updating the SE population. We show empirically that our method is capable of learning SEs for two discrete-action-space tasks (CartPole-v0 and Acrobot-v1) that allow us to train agents more robustly and with up to 60% fewer steps. Not only do we show in experiments with 4000 evaluations that the SEs are robust against hyperparameter changes such as the learning rate, batch sizes and network sizes, we also show that SEs trained with DDQN agents transfer in limited ways to a discrete-action-space version of TD3 and very well to Dueling DDQN.

📄 PDF Abstract BibTeX arXiv:2101.09721

Code (1)

automl/learning_environments 공식 구현 pytorch

Tasks

Acrobotreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

ReLU How Do I Communicate to Expedia? How Do I Communicate to Expedia? – Call ☎️ +1-(888) 829 (0881) or +1-805-330-4056 or +1-805-330-4056 for Live Support & Special Travel…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Adam 설명 없음

Target Policy Smoothing Target Policy Smoothing is a regularization strategy for the value function in reinforcement learning. Deterministic policies can overfit to narrow peaks in the value…

Experience Replay Experience Replay is a replay memory technique used in reinforcement learning where we store the agent’s experiences at each time-step, $e\_{t} = \left(s\_{t}, a\_{t}, r\_{t},…

Clipped Double Q-learning 설명 없음

TD3 TD3 builds on the DDPG algorithm for reinforcement learning, with a couple of modifications aimed at tackling overestimation bias…

Synthetic Counteradaptation: A Principle of Human-AI Co-evolution

2026-03-31 · Ivar Frisch, Jackie Kay, Philip Moreira Tomei arxiv

In this paper, we introduce the concept of synthetic counteradaptation, a process where human and AI systems co-evolve by adapting to each other's strategies and behaviors. Synthetic counteradaptation occurs when AI syst…

Game of Go

Instance Weighted Incremental Evolution Strategies for Reinforcement Learning in Dynamic Environments

2020-10-09 · Zhi Wang, Chunlin Chen, Daoyi Dong

Evolution strategies (ES), as a family of black-box optimization algorithms, recently emerge as a scalable alternative to reinforcement learning (RL) approaches such as Q-learning or policy gradient, and are much faster …

Incremental Learning Q-Learning reinforcement-learning Reinforcement Learning (RL)+1

Evolutionary Strategy Guided Reinforcement Learning via MultiBuffer Communication

2023-06-20 · Adam Callaghan, Karl Mason, Patrick Mannion

Evolutionary Algorithms and Deep Reinforcement Learning have both successfully solved control problems across a variety of domains. Recently, algorithms have been proposed which combine these two methods, aiming to lever…

Deep Reinforcement Learning Evolutionary Algorithms MuJoCo reinforcement-learning+1

Deep Reinforcement Learning Versus Evolution Strategies: A Comparative Survey

2021-09-28 · Amjad Yousef Majid, Serge Saaybi, Tomas van Rietbergen, Vincent Francois-Lavet 외

Deep Reinforcement Learning (DRL) and Evolution Strategies (ESs) have surpassed human-level control in many sequential decision-making problems, yet many open challenges still exist. To get insights into the strengths an…

Decision Making Deep Reinforcement Learning reinforcement-learning Reinforcement Learning+3

Solving Deep Reinforcement Learning Tasks with Evolution Strategies and Linear Policy Networks

2024-02-10 · Annie Wong, Jacob de Nobel, Thomas Bäck, Aske Plaat 외

Although deep reinforcement learning methods can learn effective policies for challenging problems such as Atari games and robotics tasks, algorithms are complex, and training times are often long. This study investigate…

Atari Games Deep Reinforcement Learning Q-Learning reinforcement-learning+1