State-Aware Proximal Pessimistic Algorithms for Offline Reinforcement Learning

2022-11-28 · Chen Chen, Hongyao Tang, Yi Ma, Chao Wang, Qianli Shen, Dong Li, Jianye Hao

Pessimism is of great importance in offline reinforcement learning (RL). One broad category of offline RL algorithms fulfills pessimism by explicit or implicit behavior regularization. However, most of them only consider policy divergence as behavior regularization, ignoring the effect of how the offline state distribution differs with that of the learning policy, which may lead to under-pessimism for some states and over-pessimism for others. Taking account of this problem, we propose a principled algorithmic framework for offline RL, called \emph{State-Aware Proximal Pessimism} (SA-PP). The key idea of SA-PP is leveraging discounted stationary state distribution ratios between the learning policy and the offline dataset to modulate the degree of behavior regularization in a state-wise manner, so that pessimism can be implemented in a more appropriate way. We first provide theoretical justifications on the superiority of SA-PP over previous algorithms, demonstrating that SA-PP produces a lower suboptimality upper bound in a broad range of settings. Furthermore, we propose a new algorithm named \emph{State-Aware Conservative Q-Learning} (SA-CQL), by building SA-PP upon representative CQL algorithm with the help of DualDICE for estimating discounted stationary state distribution ratios. Extensive experiments on standard offline RL benchmark show that SA-CQL outperforms the popular baselines on a large portion of benchmarks and attains the highest average return.

📄 PDF Abstract BibTeX arXiv:2211.15065

Code (0)

등록된 구현이 없습니다.

Tasks

Offline RLQ-Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

POPO: Pessimistic Offline Policy Optimization

2020-12-26 · Qiang He, Xinwen Hou

Offline reinforcement learning (RL), also known as batch RL, aims to optimize policy from a large pre-recorded dataset without interaction with the environment. This setting offers the promise of utilizing diverse, pre-c…

Offline RL Q-Learning Reinforcement Learning (RL)

Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning

2022-02-23 · ICLR 2022 4 · Chenjia Bai, Lingxiao Wang, Zhuoran Yang, Zhihong Deng 외

Offline Reinforcement Learning (RL) aims to learn policies from previously collected datasets without exploring the environment. Directly applying off-policy algorithms to offline RL usually fails due to the extrapolatio…

D4RL Offline RL reinforcement-learning Reinforcement Learning+2

Feasibility-Aware Pessimistic Estimation: Toward Long-Horizon Safety in Offline RL

2025-05-13 · Zhikun Tao, Gang Xiong, He Fang, Zhen Shen 외

Offline safe reinforcement learning(OSRL) derives constraint-satisfying policies from pre-collected datasets, offers a promising avenue for deploying RL in safety-critical real-world domains such as robotics. However, th…

Offline RL Safe Reinforcement Learning

Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexity

2022-02-28 · Laixi Shi, Gen Li, Yuting Wei, Yuxin Chen 외

Offline or batch reinforcement learning seeks to learn a near-optimal policy using history data without active exploration of the environment. To counter the insufficient coverage and sample scarcity of many offline data…

Offline RL Q-Learning reinforcement-learning Reinforcement Learning (RL)

Pessimism in the Face of Confounders: Provably Efficient Offline Reinforcement Learning in Partially Observable Markov Decision Processes

2022-05-26 · Miao Lu, Yifei Min, Zhaoran Wang, Zhuoran Yang

We study offline reinforcement learning (RL) in partially observable Markov decision processes. In particular, we aim to learn an optimal policy from a dataset collected by a behavior policy which possibly depends on the…

Causal Inference Offline RL Reinforcement Learning (RL)