Dealing with the Unknown: Pessimistic Offline Reinforcement Learning

2021-11-09 · Jinning Li, Chen Tang, Masayoshi Tomizuka, Wei Zhan

Reinforcement Learning (RL) has been shown effective in domains where the agent can learn policies by actively interacting with its operating environment. However, if we change the RL scheme to offline setting where the agent can only update its policy via static datasets, one of the major issues in offline reinforcement learning emerges, i.e. distributional shift. We propose a Pessimistic Offline Reinforcement Learning (PessORL) algorithm to actively lead the agent back to the area where it is familiar by manipulating the value function. We focus on problems caused by out-of-distribution (OOD) states, and deliberately penalize high values at states that are absent in the training dataset, so that the learned pessimistic value function lower bounds the true value anywhere within the state space. We evaluate the PessORL algorithm on various benchmark tasks, where we show that our method gains better performance by explicitly handling OOD states, when compared to those methods merely considering OOD actions.

📄 PDF Abstract BibTeX arXiv:2111.05440

Code (0)

등록된 구현이 없습니다.

Tasks

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Pessimistic Auxiliary Policy for Offline Reinforcement Learning

2026-02-27 · Fan Zhang, Baoru Huang, Xin Zhang arxiv

Offline reinforcement learning aims to learn an agent from pre-collected datasets, avoiding unsafe and inefficient real-time interaction. However, inevitable access to out-ofdistribution actions during the learning proce…

Reinforcement Learning Offline RL

Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage

2021-07-13 · ICLR 2022 4 · Masatoshi Uehara, Wen Sun

We study model-based offline Reinforcement Learning with general function approximation without a full coverage assumption on the offline data distribution. We present an algorithm named Constrained Pessimistic Policy Op…

Offline RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1

MOReL: Model-Based Offline Reinforcement Learning

2020-12-01 · NeurIPS 2020 12 · Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, Thorsten Joachims

In offline reinforcement learning (RL), the goal is to learn a highly rewarding policy based solely on a dataset of historical interactions with the environment. This serves as an extreme test for an agent's ability to e…

model Offline RL reinforcement-learning Reinforcement Learning+1

Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning

2022-02-23 · ICLR 2022 4 · Chenjia Bai, Lingxiao Wang, Zhuoran Yang, Zhihong Deng 외

Offline Reinforcement Learning (RL) aims to learn policies from previously collected datasets without exploring the environment. Directly applying off-policy algorithms to offline RL usually fails due to the extrapolatio…

D4RL Offline RL reinforcement-learning Reinforcement Learning+2

Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexity

2022-02-28 · Laixi Shi, Gen Li, Yuting Wei, Yuxin Chen 외

Offline or batch reinforcement learning seeks to learn a near-optimal policy using history data without active exploration of the environment. To counter the insufficient coverage and sample scarcity of many offline data…

Offline RL Q-Learning reinforcement-learning Reinforcement Learning (RL)