On the Role of Discount Factor in Offline Reinforcement Learning

2022-06-07 · Hao Hu, Yiqin Yang, Qianchuan Zhao, Chongjie Zhang

Offline reinforcement learning (RL) enables effective learning from previously collected data without exploration, which shows great promise in real-world applications when exploration is expensive or even infeasible. The discount factor, $\gamma$, plays a vital role in improving online RL sample efficiency and estimation accuracy, but the role of the discount factor in offline RL is not well explored. This paper examines two distinct effects of $\gamma$ in offline RL with theoretical analysis, namely the regularization effect and the pessimism effect. On the one hand, $\gamma$ is a regulator to trade-off optimality with sample efficiency upon existing offline techniques. On the other hand, lower guidance $\gamma$ can also be seen as a way of pessimism where we optimize the policy's performance in the worst possible models. We empirically verify the above theoretical observation with tabular MDPs and standard D4RL tasks. The results show that the discount factor plays an essential role in the performance of offline RL algorithms, both under small data regimes upon existing offline methods and in large data regimes without other conservative methods.

📄 PDF Abstract BibTeX arXiv:2206.03383

Code (0)

등록된 구현이 없습니다.

Tasks

D4RL Offline RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Decoupling Time and Risk: Risk-Sensitive Reinforcement Learning with General Discounting

2026-02-04 · Mehrdad Moghimi, Anthony Coache, Hyejin Ku arxiv

Distributional reinforcement learning (RL) is a powerful framework increasingly adopted in safety-critical domains for its ability to optimize risk-sensitive objectives. However, the role of the discount factor is often …

Reinforcement Learning

Offline Reinforcement Learning with Realizability and Single-policy Concentrability

2022-02-09 · Wenhao Zhan, Baihe Huang, Audrey Huang, Nan Jiang 외

Sample-efficiency guarantees for offline reinforcement learning (RL) often rely on strong assumptions on both the function classes (e.g., Bellman-completeness) and the data coverage (e.g., all-policy concentrability). De…

Offline RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

A Primal-Dual Algorithm for Offline Constrained Reinforcement Learning with Linear MDPs

2024-02-07 · Kihyuk Hong, Ambuj Tewari

We study offline reinforcement learning (RL) with linear MDPs under the infinite-horizon discounted setting which aims to learn a policy that maximizes the expected discounted cumulative reward using a pre-collected data…

Offline RL Reinforcement Learning (RL)

How to Discount Deep Reinforcement Learning: Towards New Dynamic Strategies

2015-12-07 · Vincent François-Lavet, Raphael Fonteneau, Damien Ernst

Using deep neural nets as function approximator for reinforcement learning tasks have recently been shown to be very powerful for solving problems approaching real-world complexity. Using these results as a benchmark, we…

Deep Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Near-Optimal Offline Reinforcement Learning via Double Variance Reduction

2021-02-02 · NeurIPS 2021 12 · Ming Yin, Yu Bai, Yu-Xiang Wang

We consider the problem of offline reinforcement learning (RL) -- a well-motivated setting of RL that aims at policy optimization using only historical data. Despite its wide applicability, theoretical understandings of …