Pessimistic Model Selection for Offline Deep Reinforcement Learning

2021-11-29 · Chao-Han Huck Yang, Zhengling Qi, Yifan Cui, Pin-Yu Chen

Deep Reinforcement Learning (DRL) has demonstrated great potentials in solving sequential decision making problems in many applications. Despite its promising performance, practical gaps exist when deploying DRL in real-world scenarios. One main barrier is the over-fitting issue that leads to poor generalizability of the policy learned by DRL. In particular, for offline DRL with observational data, model selection is a challenging task as there is no ground truth available for performance demonstration, in contrast with the online setting with simulated environments. In this work, we propose a pessimistic model selection (PMS) approach for offline DRL with a theoretical guarantee, which features a provably effective framework for finding the best policy among a set of candidate models. Two refined approaches are also proposed to address the potential bias of DRL model in identifying the optimal policy. Numerical studies demonstrated the superior performance of our approach over existing methods.

📄 PDF Abstract BibTeX arXiv:2111.14346

Code (0)

등록된 구현이 없습니다.

Tasks

Decision MakingDeep Reinforcement LearningModel Selectionreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Sequential Decision Making

Pessimistic Auxiliary Policy for Offline Reinforcement Learning

2026-02-27 · Fan Zhang, Baoru Huang, Xin Zhang arxiv

Offline reinforcement learning aims to learn an agent from pre-collected datasets, avoiding unsafe and inefficient real-time interaction. However, inevitable access to out-ofdistribution actions during the learning proce…

Reinforcement Learning Offline RL

MOReL: Model-Based Offline Reinforcement Learning

2020-12-01 · NeurIPS 2020 12 · Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, Thorsten Joachims

In offline reinforcement learning (RL), the goal is to learn a highly rewarding policy based solely on a dataset of historical interactions with the environment. This serves as an extreme test for an agent's ability to e…

model Offline RL reinforcement-learning Reinforcement Learning+1

Pessimistic Value Iteration for Multi-Task Data Sharing in Offline Reinforcement Learning

2024-04-30 · Chenjia Bai, Lingxiao Wang, Jianye Hao, Zhuoran Yang 외

Offline Reinforcement Learning (RL) has shown promising results in learning a task-specific policy from a fixed dataset. However, successful offline RL often relies heavily on the coverage and quality of the given datase…

Offline RL Reinforcement Learning (RL)Uncertainty Quantification

Dealing with the Unknown: Pessimistic Offline Reinforcement Learning

2021-11-09 · Jinning Li, Chen Tang, Masayoshi Tomizuka, Wei Zhan

Reinforcement Learning (RL) has been shown effective in domains where the agent can learn policies by actively interacting with its operating environment. However, if we change the RL scheme to offline setting where the …

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning

2022-02-23 · ICLR 2022 4 · Chenjia Bai, Lingxiao Wang, Zhuoran Yang, Zhihong Deng 외

Offline Reinforcement Learning (RL) aims to learn policies from previously collected datasets without exploring the environment. Directly applying off-policy algorithms to offline RL usually fails due to the extrapolatio…

D4RL Offline RL reinforcement-learning Reinforcement Learning+2