Pessimistic Value Iteration for Multi-Task Data Sharing in Offline Reinforcement Learning

2024-04-30 · Chenjia Bai, Lingxiao Wang, Jianye Hao, Zhuoran Yang, Bin Zhao, Zhen Wang, Xuelong Li

Offline Reinforcement Learning (RL) has shown promising results in learning a task-specific policy from a fixed dataset. However, successful offline RL often relies heavily on the coverage and quality of the given dataset. In scenarios where the dataset for a specific task is limited, a natural approach is to improve offline RL with datasets from other tasks, namely, to conduct Multi-Task Data Sharing (MTDS). Nevertheless, directly sharing datasets from other tasks exacerbates the distribution shift in offline RL. In this paper, we propose an uncertainty-based MTDS approach that shares the entire dataset without data selection. Given ensemble-based uncertainty quantification, we perform pessimistic value iteration on the shared offline dataset, which provides a unified framework for single- and multi-task offline RL. We further provide theoretical analysis, which shows that the optimality gap of our method is only related to the expected data coverage of the shared dataset, thus resolving the distribution shift issue in data sharing. Empirically, we release an MTDS benchmark and collect datasets from three challenging domains. The experimental results show our algorithm outperforms the previous state-of-the-art methods in challenging MTDS problems. See https://github.com/Baichenjia/UTDS for the datasets and code.

📄 PDF Abstract BibTeX arXiv:2404.19346

Code (1)

baichenjia/utds 공식 구현 pytorch

Tasks

Offline RL Reinforcement Learning (RL)Uncertainty Quantification

Pessimistic Nonlinear Least-Squares Value Iteration for Offline Reinforcement Learning

2023-10-02 · Qiwei Di, Heyang Zhao, Jiafan He, Quanquan Gu

Offline reinforcement learning (RL), where the agent aims to learn the optimal policy based on the data collected by a behavior policy, has attracted increasing attention in recent years. While offline RL with linear fun…

Offline RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Pessimistic Minimax Value Iteration: Provably Efficient Equilibrium Learning from Offline Datasets

2022-02-15 · Han Zhong, Wei Xiong, Jiyuan Tan, LiWei Wang 외

We study episodic two-player zero-sum Markov games (MGs) in the offline setting, where the goal is to find an approximate Nash equilibrium (NE) policy pair based on a dataset collected a priori. When the dataset does not…

Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexity

2022-02-28 · Laixi Shi, Gen Li, Yuting Wei, Yuxin Chen 외

Offline or batch reinforcement learning seeks to learn a near-optimal policy using history data without active exploration of the environment. To counter the insufficient coverage and sample scarcity of many offline data…

Offline RL Q-Learning reinforcement-learning Reinforcement Learning (RL)

Offline Constrained Multi-Objective Reinforcement Learning via Pessimistic Dual Value Iteration

2021-12-01 · NeurIPS 2021 12 · Runzhe Wu, Yufeng Zhang, Zhuoran Yang, Zhaoran Wang

In constrained multi-objective RL, the goal is to learn a policy that achieves the best performance specified by a multi-objective preference function under a constraint. We focus on the offline setting where the RL agen…

Multi-Objective Reinforcement Learning reinforcement-learning Reinforcement Learning (RL)

A Contracting Dynamical System Perspective toward Interval Markov Decision Processes

2023-09-17 · Saber Jafarpour, Samuel Coogan

Interval Markov decision processes are a class of Markov models where the transition probabilities between the states belong to intervals. In this paper, we study the problem of efficient estimation of the optimal polici…