Sample-Efficient Policy Space Response Oracles with Joint Experience Best Response

2026-02-06 · Ariyan Bighashdel, Thiago D. Simão, Frans A. Oliehoek arxiv

Multi-agent reinforcement learning (MARL) offers a scalable alternative to exact game-theoretic analysis but suffers from non-stationarity and the need to maintain diverse populations of strategies that capture non-transitive interactions. Policy Space Response Oracles (PSRO) address these issues by iteratively expanding a restricted game with approximate best responses (BRs), yet per-agent BR training makes it prohibitively expensive in many-agent or simulator-expensive settings. We introduce Joint Experience Best Response (JBR), a drop-in modification to PSRO that collects trajectories once under the current meta-strategy profile and reuses this joint dataset to compute BRs for all agents simultaneously. This amortizes environment interaction and improves the sample efficiency of best-response computation. Because JBR converts BR computation into an offline RL problem, we propose three remedies for distribution-shift bias: (i) Conservative JBR with safe policy improvement, (ii) Exploration-Augmented JBR that perturbs data collection and admits theoretical guarantees, and (iii) Hybrid BR that interleaves JBR with periodic independent BR updates. Across benchmark multi-agent environments, Exploration-Augmented JBR achieves the best accuracy-efficiency trade-off, while Hybrid BR attains near-PSRO performance at a fraction of the sample cost. Overall, JBR makes PSRO substantially more practical for large-scale strategic learning while preserving equilibrium robustness.

📄 PDF Abstract BibTeX arXiv:2602.06599

Code (0)

등록된 구현이 없습니다.

Tasks

Multi-agent Reinforcement Learning Offline RL

Code-Space Response Oracles: Generating Interpretable Multi-Agent Policies with Large Language Models

2026-03-10 · Daniel Hennes, Zun Li, John Schultz, Marc Lanctot arxiv

Recent advances in multi-agent reinforcement learning, particularly Policy-Space Response Oracles (PSRO), have enabled the computation of approximate game-theoretic equilibria in increasingly complex domains. However, th…

Multi-agent Reinforcement Learning Code Generation

Fusion-PSRO: Nash Policy Fusion for Policy Space Response Oracles

2024-05-31 · Jiesong Lian, Yucong Huang, Chengdong Ma, Mingzhi Wang 외

For solving zero-sum games involving non-transitivity, a useful approach is to maintain a policy population to approximate the Nash Equilibrium (NE). Previous studies have shown that the Policy Space Response Oracles (PS…

Multi-agent Reinforcement Learning

Policy Space Diversity for Non-Transitive Games

2023-06-29 · NeurIPS 2023 11

Policy-Space Response Oracles (PSRO) is an influential algorithm framework for approximating a Nash Equilibrium (NE) in multi-agent non-transitive games. Many previous studies have been trying to promote policy diversity…

Diversity

Policy Space Response Oracles: A Survey

2024-03-04 · Ariyan Bighashdel, Yongzhao Wang, Stephen Mcaleer, Rahul Savani 외

Game theory provides a mathematical way to study the interaction between multiple decision makers. However, classical game-theoretic analysis is limited in scalability due to the large number of strategies, precluding di…

Position Survey

Global Policy-Space Response Oracles for Two-Player Zero-Sum Games

2026-05-27 · Junyu Zhang, Feihong Yang, Jian Wang, Chao Wang 외 arxiv

The Policy-Space Response Oracles (PSRO) framework scales equilibrium computation to large zero-sum games by iteratively expanding a restricted strategy set using deep reinforcement learning (DRL). A central challenge is…

Reinforcement Learning