Instance Weighted Incremental Evolution Strategies for Reinforcement Learning in Dynamic Environments
Evolution strategies (ES), as a family of black-box optimization algorithms, recently emerge as a scalable alternative to reinforcement learning (RL) approaches such as Q-learning or policy gradient, and are much faster when many central processing units (CPUs) are available due to better parallelization. In this paper, we propose a systematic incremental learning method for ES in dynamic environments. The goal is to adjust previously learned policy to a new one incrementally whenever the environment changes. We incorporate an instance weighting mechanism with ES to facilitate its learning adaptation, while retaining scalability of ES. During parameter updating, higher weights are assigned to instances that contain more new knowledge, thus encouraging the search distribution to move towards new promising areas of parameter space. We propose two easy-to-implement metrics to calculate the weights: instance novelty and instance quality. Instance novelty measures an instance's difference from the previous optimum in the original environment, while instance quality corresponds to how well an instance performs in the new environment. The resulting algorithm, Instance Weighted Incremental Evolution Strategies (IW-IES), is verified to achieve significantly improved performance on challenging RL tasks ranging from robot navigation to locomotion. This paper thus introduces a family of scalable ES algorithms for RL domains that enables rapid learning adaptation to dynamic environments.
Code (1)
Tasks
Incremental LearningQ-Learningreinforcement-learningReinforcement Learning (RL)Robot NavigationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Importance Weighted Evolution Strategies
Evolution Strategies (ES) emerged as a scalable alternative to popular Reinforcement Learning (RL) techniques, providing an almost perfect speedup when distributed across hundreds of CPU cores thanks to a reduced communi…
CPUreinforcement-learningReinforcement LearningReinforcement Learning (RL)The Strategy Evolution in Double Auction Based on the Experience-Weighted Attraction Learning Model
The double auction is a widely applicable trading mechanism used to converge to competitive equilibrium in different markets from which multiple equilibriums and incomplete information may arise. Therefore, different lea…
Incremental Constrained Clustering by Minimal Weighted Modification
Clustering is a well-known task in Data Mining that aims at grouping data instances according to their similarity. It is an exploratory and unsupervised task whose results depend on many parameters, often requiring the e…
ClusteringConstrained ClusteringIncremental Constrained ClusteringTime SeriesA Globally Convergent Evolutionary Strategy for Stochastic Constrained Optimization with Applications to Reinforcement Learning
Evolutionary strategies have recently been shown to achieve competing levels of performance for complex optimization problems in reinforcement learning. In such problems, one often needs to optimize an objective function…
Solving Deep Reinforcement Learning Tasks with Evolution Strategies and Linear Policy Networks
Although deep reinforcement learning methods can learn effective policies for challenging problems such as Atari games and robotics tasks, algorithms are complex, and training times are often long. This study investigate…
Atari GamesDeep Reinforcement LearningQ-Learningreinforcement-learning+1