Sample Complexity of Offline Distributionally Robust Linear Markov Decision Processes

2024-03-19 · He Wang, Laixi Shi, Yuejie Chi

In offline reinforcement learning (RL), the absence of active exploration calls for attention on the model robustness to tackle the sim-to-real gap, where the discrepancy between the simulated and deployed environments can significantly undermine the performance of the learned policy. To endow the learned policy with robustness in a sample-efficient manner in the presence of high-dimensional state-action space, this paper considers the sample complexity of distributionally robust linear Markov decision processes (MDPs) with an uncertainty set characterized by the total variation distance using offline data. We develop a pessimistic model-based algorithm and establish its sample complexity bound under minimal data coverage assumptions, which outperforms prior art by at least $\widetilde{O}(d)$, where $d$ is the feature dimension. We further improve the performance guarantee of the proposed algorithm by incorporating a carefully-designed variance estimator.

📄 PDF Abstract BibTeX arXiv:2403.12946

Code (0)

등록된 구현이 없습니다.

Tasks

Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

SET Dynamic Sparse Training method where weight mask is updated randomly periodically

Distributionally Robust Model-Based Offline Reinforcement Learning with Near-Optimal Sample Complexity

2022-08-11 · Laixi Shi, Yuejie Chi

This paper concerns the central issues of model robustness and sample efficiency in offline reinforcement learning (RL), which aims to learn to perform decision making from history data without active exploration. Due to…

Decision Making Offline RL reinforcement-learning Reinforcement Learning (RL)

Bridging Distributionally Robust Learning and Offline RL: An Approach to Mitigate Distribution Shift and Partial Data Coverage

2023-10-27 · Kishan Panaganti, Zaiyan Xu, Dileep Kalathil, Mohammad Ghavamzadeh

The goal of an offline reinforcement learning (RL) algorithm is to learn optimal polices using historical (offline) data, without access to the environment for online exploration. One of the main challenges in offline RL…

Offline RL Reinforcement Learning (RL)

Robust Offline Reinforcement Learning for Non-Markovian Decision Processes

2024-11-12 · Ruiquan Huang, Yingbin Liang, Jing Yang

Distributionally robust offline reinforcement learning (RL) aims to find a policy that performs the best under the worst environment within an uncertainty set using an offline dataset collected from a nominal model. Whil…

Dataset Distillation reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Distributionally Robust Offline Reinforcement Learning with Linear Function Approximation

2022-09-14 · Xiaoteng Ma, Zhipeng Liang, Jose Blanchet, Mingwen Liu 외

Among the reasons hindering reinforcement learning (RL) applications to real-world problems, two factors are critical: limited data and the mismatch between the testing environment (real environment in which the policy i…

Offline RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Distributionally Robust Model-based Reinforcement Learning with Large State Spaces

2023-09-05 · Shyam Sundhar Ramesh, Pier Giuseppe Sessa, Yifan Hu, Andreas Krause 외

Three major challenges in reinforcement learning are the complex dynamical systems with large state spaces, the costly data acquisition processes, and the deviation of real-world dynamics from the training environment de…

Gaussian Processes Model-based Reinforcement Learning reinforcement-learning Reinforcement Learning