Macro action selection with deep reinforcement learning in StarCraft

2018-12-02 · Sijia Xu, Hongyu Kuang, Zhi Zhuang, Renjie Hu, Yang Liu, Huyang Sun

StarCraft (SC) is one of the most popular and successful Real Time Strategy (RTS) games. In recent years, SC is also widely accepted as a challenging testbed for AI research because of its enormous state space, partially observed information, multi-agent collaboration, and so on. With the help of annual AIIDE and CIG competitions, a growing number of SC bots are proposed and continuously improved. However, a large gap remains between the top-level bot and the professional human player. One vital reason is that current SC bots mainly rely on predefined rules to select macro actions during their games. These rules are not scalable and efficient enough to cope with the enormous yet partially observed state space in the game. In this paper, we propose a deep reinforcement learning (DRL) framework to improve the selection of macro actions. Our framework is based on the combination of the Ape-X DQN and the Long-Short-Term-Memory (LSTM). We use this framework to build our bot, named as LastOrder. Our evaluation, based on training against all bots from the AIIDE 2017 StarCraft AI competition set, shows that LastOrder achieves an 83% winning rate, outperforming 26 bots in total 28 entrants.

📄 PDF Abstract BibTeX arXiv:1812.00336

Code (1)

Bilibili/LastOrder 공식 구현 tf

Tasks

Deep Reinforcement LearningReal-Time Strategy Gamesreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Starcraft

Methods 이 논문이 사용한 방법론

Double Q-learning Double Q-learning is an off-policy reinforcement learning algorithm that utilises double estimation to counteract overestimation problems with traditional Q-learning. The…

Prioritized Experience Replay Prioritized Experience Replay is a type of experience replay in reinforcement learning where we more frequently replay…

Dueling Network A Dueling Network is a type of Q-Network that has two streams to separately estimate (scalar) state-value and the advantages for each action. Both streams share a common…

N-step Returns $n$-step Returns are used for value function estimation in reinforcement learning. Specifically, for $n$ steps we can write the complete return as: $$ R\_{t}^{(n)} =…

Ape-X Ape-X is a distributed architecture for deep reinforcement learning. The algorithm decouples acting from learning: the actors interact with their own instances of the…

Ape-X DQN Ape-X DQN is a variant of a DQN with some components of Rainbow-DQN that utilizes…

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Modular Architecture for StarCraft II with Deep Reinforcement Learning

2018-11-08 · Dennis Lee, Haoran Tang, Jeffrey O. Zhang, Huazhe Xu 외

We present a novel modular architecture for StarCraft II AI. The architecture splits responsibilities between multiple modules that each control one aspect of the game, such as build-order selection or tactics. A central…

Deep Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+2

Learning Macromanagement in StarCraft from Replays using Deep Learning

2017-07-12 · Niels Justesen, Sebastian Risi

The real-time strategy game StarCraft has proven to be a challenging environment for artificial intelligence techniques, and as a result, current state-of-the-art solutions consist of numerous hand-crafted modules. In th…

Deep Learning Deep Reinforcement Learning Reinforcement Learning Starcraft

On Reinforcement Learning for Full-length Game of StarCraft

2018-09-23 · Zhen-Jia Pang, Ruo-Ze Liu, Zhou-Yu Meng, Yi Zhang 외

StarCraft II poses a grand challenge for reinforcement learning. The main difficulties of it include huge state and action space and a long-time horizon. In this paper, we investigate a hierarchical reinforcement learnin…

CPU Hierarchical Reinforcement Learning reinforcement-learning Reinforcement Learning+4

StarCraft II: A New Challenge for Reinforcement Learning

2017-08-16 · Oriol Vinyals, Timo Ewalds, Sergey Bartunov, Petko Georgiev 외

This paper introduces SC2LE (StarCraft II Learning Environment), a reinforcement learning environment based on the StarCraft II game. This domain poses a new grand challenge for reinforcement learning, representing a mor…

Deep Reinforcement Learning Real-Time Strategy Games reinforcement-learning Reinforcement Learning+3

MSC: A Dataset for Macro-Management in StarCraft II

2017-10-09 · Huikai Wu, Yanqi Zong, Junge Zhang, Kaiqi Huang

Macro-management is an important problem in StarCraft, which has been studied for a long time. Various datasets together with assorted methods have been proposed in the last few years. But these datasets have some defect…

Management Real-Time Strategy Games Starcraft Starcraft II