Offline Equilibrium Finding

2022-07-12 · Shuxin Li, Xinrun Wang, Youzhi Zhang, Jakub Cerny, Pengdeng Li, Hau Chan, Bo An

Offline reinforcement learning (offline RL) is an emerging field that has recently begun gaining attention across various application domains due to its ability to learn strategies from earlier collected datasets. Offline RL proved very successful, paving a path to solving previously intractable real-world problems, and we aim to generalize this paradigm to a multiplayer-game setting. To this end, we introduce a problem of offline equilibrium finding (OEF) and construct multiple types of datasets across a wide range of games using several established methods. To solve the OEF problem, we design a model-based framework that can directly apply any online equilibrium finding algorithm to the OEF setting while making minimal changes. The three most prominent contemporary online equilibrium finding algorithms are adapted to the context of OEF, creating three model-based variants: OEF-PSRO and OEF-CFR, which generalize the widely-used algorithms PSRO and Deep CFR to compute Nash equilibria (NEs), and OEF-JPSRO, which generalizes the JPSRO to calculate (Coarse) Correlated equilibria ((C)CEs). We also combine the behavior cloning policy with the model-based policy to further improve the performance and provide a theoretical guarantee of the solution quality. Extensive experimental results demonstrate the superiority of our approach over offline RL algorithms and the importance of using model-based methods for OEF problems. We hope our work will contribute to advancing research in large-scale equilibrium finding.

📄 PDF Abstract BibTeX arXiv:2207.05285

Code (1)

securitygames/oef 공식 구현 pytorch

Tasks

Offline RL

Pessimistic Minimax Value Iteration: Provably Efficient Equilibrium Learning from Offline Datasets

2022-02-15 · Han Zhong, Wei Xiong, Jiyuan Tan, LiWei Wang 외

We study episodic two-player zero-sum Markov games (MGs) in the offline setting, where the goal is to find an approximate Nash equilibrium (NE) policy pair based on a dataset collected a priori. When the dataset does not…

Pessimism-Free Offline Learning in General-Sum Games via KL Regularization

2026-04-30 · Claire Chen, Yuheng Zhang arxiv

Offline multi-agent reinforcement learning in general-sum settings is challenged by the distribution shift between logged datasets and target equilibrium policies. While standard methods rely on manual pessimistic penalt…

Multi-agent Reinforcement Learning

Conservative Equilibrium Discovery in Offline Game-Theoretic Multiagent Reinforcement Learning

2026-02-27 · Austin A. Nguyen, Michael P. Wellman arxiv

Offline learning of strategies takes data efficiency to its extreme by restricting algorithms to a fixed dataset of state-action trajectories. We consider the problem in a mixed-motive multiagent setting, where the goal …

Reinforcement Learning

Learning Zero-Sum Simultaneous-Move Markov Games Using Function Approximation and Correlated Equilibrium

2020-02-17 · Qiaomin Xie, Yudong Chen, Zhaoran Wang, Zhuoran Yang

We develop provably efficient reinforcement learning algorithms for two-player zero-sum finite-horizon Markov games with simultaneous moves. To incorporate function approximation, we consider a family of Markov games whe…

Reinforcement Learning

Offline Learning in Markov Games with General Function Approximation

2023-02-06 · Yuheng Zhang, Yu Bai, Nan Jiang

We study offline multi-agent reinforcement learning (RL) in Markov games, where the goal is to learn an approximate equilibrium -- such as Nash equilibrium and (Coarse) Correlated Equilibrium -- from an offline dataset p…

Multi-agent Reinforcement Learning Reinforcement Learning (RL)