WarpDrive: Extremely Fast End-to-End Deep Multi-Agent Reinforcement Learning on a GPU
Deep reinforcement learning (RL) is a powerful framework to train decision-making models in complex environments. However, RL can be slow as it requires repeated interaction with a simulation of the environment. In particular, there are key system engineering bottlenecks when using RL in complex environments that feature multiple agents with high-dimensional state, observation, or action spaces. We present WarpDrive, a flexible, lightweight, and easy-to-use open-source RL framework that implements end-to-end deep multi-agent RL on a single GPU (Graphics Processing Unit), built on PyCUDA and PyTorch. Using the extreme parallelization capability of GPUs, WarpDrive enables orders-of-magnitude faster RL compared to common implementations that blend CPU simulations and GPU models. Our design runs simulations and the agents in each simulation in parallel. It eliminates data copying between CPU and GPU. It also uses a single simulation data store on the GPU that is safely updated in-place. WarpDrive provides a lightweight Python interface and flexible environment wrappers that are easy to use and extend. Together, this allows the user to easily run thousands of concurrent multi-agent simulations and train on extremely large batches of experience. Through extensive experiments, we verify that WarpDrive provides high-throughput and scales almost linearly to many agents and parallel environments. For example, WarpDrive yields 2.9 million environment steps/second with 2000 environments and 1000 agents (at least 100x higher throughput compared to a CPU implementation) in a benchmark Tag simulation. As such, WarpDrive is a fast and extensible multi-agent RL platform to significantly accelerate research and development.
Code (3)
Tasks
CPUDecision MakingDeep Reinforcement LearningGPUMulti-agent Reinforcement LearningReinforcement Learning (RL)TAGSimilar Papers 제목 키워드 기반
Scalable Centralized Deep Multi-Agent Reinforcement Learning via Policy Gradients
In this paper, we explore using deep reinforcement learning for problems with multiple agents. Most existing methods for deep multi-agent reinforcement learning consider only a small number of agents. When the number of …
Deep Reinforcement LearningDistributed OptimizationMulti-agent Reinforcement Learningreinforcement-learning+2Memory Sequence Length of Data Sampling Impacts the Adaptation of Meta-Reinforcement Learning Agents
Fast adaptation to new tasks is extremely important for embodied agents in the real world. Meta-reinforcement learning (meta-RL) has emerged as an effective method to enable fast adaptation in unknown environments. Compa…
continuous-controlContinuous ControlMeta Reinforcement LearningMuJoCo+1Divide and Explore: Multi-Agent Separate Exploration with Shared Intrinsic Motivations
One of the greatest challenges of reinforcement learning is efficient exploration, especially when training signals are sparse or deceptive. The main difficulty of exploration lies in the size and complexity of the state…
Distributed ComputingEfficient ExplorationMulti-agent Deep Reinforcement Learning with Extremely Noisy Observations
Multi-agent reinforcement learning systems aim to provide interacting agents with the ability to collaboratively learn and adapt to the behaviour of other agents. In many real-world applications, the agents can only acqu…
Deep Reinforcement LearningMulti-agent Reinforcement Learningreinforcement-learningReinforcement Learning+1Autonomous Air Traffic Controller: A Deep Multi-Agent Reinforcement Learning Approach
Air traffic control is a real-time safety-critical decision making process in highly dynamic and stochastic environments. In today's aviation practice, a human air traffic controller monitors and directs many aircraft fl…
Decision MakingMulti-agent Reinforcement Learningreinforcement-learningReinforcement Learning+1