Concurrent Meta Reinforcement Learning
State-of-the-art meta reinforcement learning algorithms typically assume the setting of a single agent interacting with its environment in a sequential manner. A negative side-effect of this sequential execution paradigm is that, as the environment becomes more and more challenging, and thus requiring more interaction episodes for the meta-learner, it needs the agent to reason over longer and longer time-scales. To combat the difficulty of long time-scale credit assignment, we propose an alternative parallel framework, which we name "Concurrent Meta-Reinforcement Learning" (CMRL), that transforms the temporal credit assignment problem into a multi-agent reinforcement learning one. In this multi-agent setting, a set of parallel agents are executed in the same environment and each of these "rollout" agents are given the means to communicate with each other. The goal of the communication is to coordinate, in a collaborative manner, the most efficient exploration of the shared task the agents are currently assigned. This coordination therefore represents the meta-learning aspect of the framework, as each agent can be assigned or assign itself a particular section of the current task's state space. This framework is in contrast to standard RL methods that assume that each parallel rollout occurs independently, which can potentially waste computation if many of the rollouts end up sampling the same part of the state space. Furthermore, the parallel setting enables us to define several reward sharing functions and auxiliary losses that are non-trivial to apply in the sequential setting. We demonstrate the effectiveness of our proposed CMRL at improving over sequential methods in a variety of challenging tasks.
Code (1)
Tasks
Efficient ExplorationMeta-LearningMeta Reinforcement LearningMulti-agent Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Sampling Attacks on Meta Reinforcement Learning: A Minimax Formulation and Complexity Analysis
Meta reinforcement learning (meta RL), as a combination of meta-learning ideas and reinforcement learning (RL), enables the agent to adapt to different tasks using a few samples. However, this sampling-based adaptation a…
Meta-LearningMeta Reinforcement Learningreinforcement-learningReinforcement Learning+1Simion Zoo: A Workbench for Distributed Experimentation with Reinforcement Learning for Continuous Control Tasks
We present Simion Zoo, a Reinforcement Learning (RL) workbench that provides a complete set of tools to design, run, and analyze the results,both statistically and visually, of RL control applications. The main features …
continuous-controlContinuous Controlreinforcement-learningReinforcement Learning+1Meta Reinforcement Learning for Optimal Design of Legged Robots
The process of robot design is a complex task and the majority of design decisions are still based on human intuition or tedious manual tuning. A more informed way of facing this task is computational design methods wher…
Meta Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)Alchemy: A benchmark and analysis toolkit for meta-reinforcement learning agents
There has been rapidly growing interest in meta-learning as a method for increasing the flexibility and sample efficiency of reinforcement learning. One problem in this area of research, however, has been a scarcity of a…
Meta-LearningMeta Reinforcement Learningreinforcement-learningReinforcement Learning (RL)+1Meta-learners' learning dynamics are unlike learners'
Meta-learning is a tool that allows us to build sample-efficient learning systems. Here we show that, once meta-trained, LSTM Meta-Learners aren't just faster learners than their sample-inefficient deep learning (DL) and…
Meta-LearningMulti-Armed BanditsregressionReinforcement Learning+1