On Efficient Reinforcement Learning for Full-length Game of StarCraft II
StarCraft II (SC2) poses a grand challenge for reinforcement learning (RL), of which the main difficulties include huge state space, varying action space, and a long time horizon. In this work, we investigate a set of RL techniques for the full-length game of StarCraft II. We investigate a hierarchical RL approach involving extracted macro-actions and a hierarchical architecture of neural networks. We investigate a curriculum transfer training procedure and train the agent on a single machine with 4 GPUs and 48 CPU threads. On a 64x64 map and using restrictive units, we achieve a win rate of 99% against the level-1 built-in AI. Through the curriculum transfer learning algorithm and a mixture of combat models, we achieve a 93% win rate against the most difficult non-cheating level built-in AI (level-7). In this extended version of the paper, we improve our architecture to train the agent against the cheating level AIs and achieve the win rate against the level-8, level-9, and level-10 AIs as 96%, 97%, and 94%, respectively. Our codes are at https://github.com/liuruoze/HierNet-SC2. To provide a baseline referring the AlphaStar for our work as well as the research and open-source community, we reproduce a scaled-down version of it, mini-AlphaStar (mAS). The latest version of mAS is 1.07, which can be trained on the raw action space which has 564 actions. It is designed to run training on a single common machine, by making the hyper-parameters adjustable. We then compare our work with mAS using the same resources and show that our method is more effective. The codes of mini-AlphaStar are at https://github.com/liuruoze/mini-AlphaStar. We hope our study could shed some light on the future research of efficient reinforcement learning on SC2 and other large-scale games.
Code (2)
Tasks
CPUreinforcement-learningReinforcement Learning (RL)StarcraftStarcraft IITransfer LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Reinforcement Learning for the Beginning of Starcraft II Game
Starcraft II is a popular real-time strategy game that is welcomed by many young people. This game is really complicated to play due to the long game timeline, various society/buildings/units, a large number of actions o…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Starcraft+1SCC: an efficient deep reinforcement learning agent mastering the game of StarCraft II
AlphaStar, the AI that reaches GrandMaster level in StarCraft II, is a remarkable milestone demonstrating what deep reinforcement learning can achieve in complex Real-Time Strategy (RTS) games. However, the complexities …
Deep Reinforcement LearningImitation Learningreinforcement-learningReinforcement Learning+3TStarBots: Defeating the Cheating Level Builtin AI in StarCraft II in the Full Game
Starcraft II (SC2) is widely considered as the most challenging Real Time Strategy (RTS) game. The underlying challenges include a large observation space, a huge (continuous and infinite) action space, partial observati…
AI AgentDecision MakingDeep Reinforcement LearningReal-Time Strategy Games+3StarCraft II: A New Challenge for Reinforcement Learning
This paper introduces SC2LE (StarCraft II Learning Environment), a reinforcement learning environment based on the StarCraft II game. This domain poses a new grand challenge for reinforcement learning, representing a mor…
Deep Reinforcement LearningReal-Time Strategy Gamesreinforcement-learningReinforcement Learning+3On Reinforcement Learning for Full-length Game of StarCraft
StarCraft II poses a grand challenge for reinforcement learning. The main difficulties of it include huge state and action space and a long-time horizon. In this paper, we investigate a hierarchical reinforcement learnin…
CPUHierarchical Reinforcement Learningreinforcement-learningReinforcement Learning+4