Sample-Efficient Tabular Self-Play for Offline Robust Reinforcement Learning

2025-11-29 · Na Li, Zewu Zheng, Wei Ni, Hangguan Shan, Wenjie Zhang, Xinyu Li arxiv

Multi-agent reinforcement learning (MARL), as a thriving field, explores how multiple agents independently make decisions in a shared dynamic environment. Due to environmental uncertainties, policies in MARL must remain robust to tackle the sim-to-real gap. We focus on robust two-player zero-sum Markov games (TZMGs) in offline settings, specifically on tabular robust TZMGs (RTZMGs). We propose a model-based algorithm (\textit{RTZ-VI-LCB}) for offline RTZMGs, which is optimistic robust value iteration combined with a data-driven Bernstein-style penalty term for robust value estimation. By accounting for distribution shifts in the historical dataset, the proposed algorithm establishes near-optimal sample complexity guarantees under partial coverage and environmental uncertainty. An information-theoretic lower bound is developed to confirm the tightness of our algorithm's sample complexity, which is optimal regarding both state and action spaces. To the best of our knowledge, RTZ-VI-LCB is the first to attain this optimality, sets a new benchmark for offline RTZMGs, and is validated experimentally.

📄 PDF Abstract BibTeX arXiv:2512.00352

Code (0)

등록된 구현이 없습니다.

Tasks

Multi-agent Reinforcement Learning

On the Role of Discount Factor in Offline Reinforcement Learning

2022-06-07 · Hao Hu, Yiqin Yang, Qianchuan Zhao, Chongjie Zhang

Offline reinforcement learning (RL) enables effective learning from previously collected data without exploration, which shows great promise in real-world applications when exploration is expensive or even infeasible. Th…

D4RL Offline RL reinforcement-learning Reinforcement Learning+1

Reward-agnostic Fine-tuning: Provable Statistical Benefits of Hybrid Reinforcement Learning

2023-05-17 · NeurIPS 2023 11

This paper studies tabular reinforcement learning (RL) in the hybrid setting, which assumes access to both an offline dataset and online interactions with the unknown environment. A central question boils down to how to …

Offline RL reinforcement-learning Reinforcement Learning (RL)

Settling the Sample Complexity of Model-Based Offline Reinforcement Learning

2022-04-11 · Gen Li, Laixi Shi, Yuxin Chen, Yuejie Chi 외

This paper is concerned with offline reinforcement learning (RL), which learns using pre-collected data without further exploration. Effective offline RL would be able to accommodate distribution shift and limited data c…

Offline RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Near-Optimal Reinforcement Learning with Self-Play

2020-06-22 · NeurIPS 2020 12 · Yu Bai, Chi Jin, Tiancheng Yu

This paper considers the problem of designing optimal algorithms for reinforcement learning in two-player zero-sum games. We focus on self-play algorithms which learn the optimal policy by playing against itself without …

Q-Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Double Pessimism is Provably Efficient for Distributionally Robust Offline Reinforcement Learning: Generic Algorithm and Robust Partial Coverage

2023-05-16 · NeurIPS 2023 11

In this paper, we study distributionally robust offline reinforcement learning (robust offline RL), which seeks to find an optimal policy purely from an offline dataset that can perform well in perturbed environments. In…

Offline RL