Distributionally Robust Model-based Reinforcement Learning with Large State Spaces

2023-09-05 · Shyam Sundhar Ramesh, Pier Giuseppe Sessa, Yifan Hu, Andreas Krause, Ilija Bogunovic

Three major challenges in reinforcement learning are the complex dynamical systems with large state spaces, the costly data acquisition processes, and the deviation of real-world dynamics from the training environment deployment. To overcome these issues, we study distributionally robust Markov decision processes with continuous state spaces under the widely used Kullback-Leibler, chi-square, and total variation uncertainty sets. We propose a model-based approach that utilizes Gaussian Processes and the maximum variance reduction algorithm to efficiently learn multi-output nominal transition dynamics, leveraging access to a generative model (i.e., simulator). We further demonstrate the statistical sample complexity of the proposed method for different uncertainty sets. These complexity bounds are independent of the number of states and extend beyond linear dynamics, ensuring the effectiveness of our approach in identifying near-optimal distributionally-robust policies. The proposed method can be further combined with other model-free distributionally robust reinforcement learning methods to obtain a near-optimal robust policy. Experimental results demonstrate the robustness of our algorithm to distributional shifts and its superior performance in terms of the number of samples needed.

📄 PDF Abstract BibTeX arXiv:2309.02236

Code (0)

등록된 구현이 없습니다.

Tasks

Gaussian ProcessesModel-based Reinforcement Learningreinforcement-learning Reinforcement Learning

Distributional Robustness and Regularization in Reinforcement Learning

2020-03-05 · Esther Derman, Shie Mannor

Distributionally Robust Optimization (DRO) has enabled to prove the equivalence between robustness and regularization in classification and regression, thus providing an analytical reason why regularization generalizes w…

Decision Making reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1

Sample Complexity of Distributionally Robust Average-Reward Reinforcement Learning

2025-05-15 · Zijun Chen, Shengbo Wang, Nian Si

Motivated by practical applications where stable long-term performance is critical-such as robotics, operations research, and healthcare-we study the problem of distributionally robust (DR) average-reward reinforcement l…

reinforcement-learning Reinforcement Learning

Distributionally Robust Imitation Learning

2021-12-01 · NeurIPS 2021 12 · Mohammad Ali Bashiri, Brian Ziebart, Xinhua Zhang

We consider the imitation learning problem of learning a policy in a Markov Decision Process (MDP) setting where the reward function is not given, but demonstrations from experts are available. Although the goal of imita…

Imitation Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Minimax Optimal and Computationally Efficient Algorithms for Distributionally Robust Offline Reinforcement Learning

2024-03-14 · Zhishuai Liu, Pan Xu

Distributionally robust offline reinforcement learning (RL), which seeks robust policy training against environment perturbation by modeling dynamics uncertainty, calls for function approximations when facing large state…

Offline RL Reinforcement Learning (RL)

Taming the Curses of Multiagency in Robust Markov Games with Large State Space through Linear Function Approximation

2026-05-04 · Jingchu Gai, Laixi Shi arxiv

Multi-agent reinforcement learning (MARL) holds great potential but faces robustness challenges due to environmental uncertainty. To address this, distributionally robust Markov games (RMGs) optimize worst-case performan…

Multi-agent Reinforcement Learning