Reinforcement Learning In Two Player Zero Sum Simultaneous Action Games

2021-10-10 · Patrick Phillips

Two player zero sum simultaneous action games are common in video games, financial markets, war, business competition, and many other settings. We first introduce the fundamental concepts of reinforcement learning in two player zero sum simultaneous action games and discuss the unique challenges this type of game poses. Then we introduce two novel agents that attempt to handle these challenges by using joint action Deep Q-Networks (DQN). The first agent, called the Best Response AgenT (BRAT), builds an explicit model of its opponent's policy using imitation learning, and then uses this model to find the best response to exploit the opponent's strategy. The second agent, Meta-Nash DQN, builds an implicit model of its opponent's policy in order to produce a context variable that is used as part of the Q-value calculation. An explicit minimax over Q-values is used to find actions close to Nash equilibrium. We find empirically that both agents converge to Nash equilibrium in a self-play setting for simple matrix games, while also performing well in games with larger state and action spaces. These novel algorithms are evaluated against vanilla RL algorithms as well as recent state of the art multi-agent and two agent algorithms. This work combines ideas from traditional reinforcement learning, game theory, and meta learning.

📄 PDF Abstract BibTeX arXiv:2110.04835

Code (1)

peweetheman/reinforcement_learning_in_two_player_simultaneous_action_games 공식 구현 pytorch

Tasks

Imitation LearningMeta-Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Vocal Bursts Valence Prediction

Methods 이 논문이 사용한 방법론

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Convolution A convolution is a type of matrix operation, consisting of a kernel, a small matrix of weights, that slides over input data performing element-wise multiplication with the…

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

DQN A DQN, or Deep Q-Network, approximates a state-value function in a Q-Learning framework with a neural network. In the Atari…

Almost Optimal Algorithms for Two-player Zero-Sum Linear Mixture Markov Games

2021-02-15 · Zixiang Chen, Dongruo Zhou, Quanquan Gu

We study reinforcement learning for two-player zero-sum Markov games with simultaneous moves in the finite-horizon setting, where the transition kernel of the underlying Markov games can be parameterized by a linear func…

Team-PSRO for Learning Approximate TMECor in Large Team Games via Cooperative Reinforcement Learning

2023-09-21 · NeurIPS 2023 11

Recent algorithms have achieved superhuman performance at a number of two-player zero-sum games such as poker and go. However, many real-world situations are multi-player games. Zero-sum two-team games, such as bridge an…

Reinforcement learning with restrictions on the action set

2013-06-12 · Mario Bravo, Mathieu Faure

Consider a 2-player normal-form game repeated over time. We introduce an adaptive learning procedure, where the players only observe their own realized payoff at each stage. We assume that agents do not know their own pa…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Learning to Play No-Press Diplomacy with Best Response Policy Iteration

2020-06-08 · NeurIPS 2020 12 · Thomas Anthony, Tom Eccles, Andrea Tacchetti, János Kramár 외

Recent advances in deep reinforcement learning (RL) have led to considerable progress in many 2-player zero-sum games, such as Go, Poker and Starcraft. The purely adversarial nature of such games allows for conceptually …

Deep Reinforcement Learning Reinforcement Learning (RL)Starcraft

Incentivize without Bonus: Provably Efficient Model-based Online Multi-agent RL for Markov Games

2025-02-13 · Tong Yang, Bo Dai, Lin Xiao, Yuejie Chi

Multi-agent reinforcement learning (MARL) lies at the heart of a plethora of applications involving the interaction of a group of agents in a shared unknown environment. A prominent framework for studying MARL is Markov …

Multi-agent Reinforcement Learning Uncertainty Quantification