Weighted model estimation for offline model-based reinforcement learning

2021-12-01 · NeurIPS 2021 12 · Toru Hishinuma, Kei Senda

This paper discusses model estimation in offline model-based reinforcement learning (MBRL), which is important for subsequent policy improvement using an estimated model. From the viewpoint of covariate shift, a natural idea is model estimation weighted by the ratio of the state-action distributions of offline data and real future data. However, estimating such a natural weight is one of the main challenges for off-policy evaluation, which is not easy to use. As an artificial alternative, this paper considers weighting with the state-action distribution ratio of offline data and simulated future data, which can be estimated relatively easily by standard density ratio estimation techniques for supervised learning. Based on the artificial weight, this paper defines a loss function for offline MBRL and presents an algorithm to optimize it. Weighting with the artificial weight is justified as evaluating an upper bound of the policy evaluation error. Numerical experiments demonstrate the effectiveness of weighting with the artificial weight.

📄 PDF Abstract BibTeX

Code (0)

등록된 구현이 없습니다.

Tasks

Density Ratio EstimationmodelModel-based Reinforcement LearningOff-policy evaluationreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning

2021-05-17 · Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua Susskind 외

Offline Reinforcement Learning promises to learn effective policies from previously-collected, static datasets without the need for exploration. However, existing Q-learning and actor-critic based off-policy RL algorithm…

Offline RL Q-Learning reinforcement-learning Reinforcement Learning+1

Uncertainty Weighted Offline Reinforcement Learning

2021-01-01 · Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua M. Susskind 외

Offline RL Q-Learning reinforcement-learning Reinforcement Learning+1

SAMG: State-Action-Aware Offline-to-Online Reinforcement Learning with Offline Model Guidance

2024-10-24 · Liyu Zhang, Haochi Wu, Xu Wan, Quan Kong 외

The offline-to-online (O2O) paradigm in reinforcement learning (RL) utilizes pre-trained models on offline datasets for subsequent online fine-tuning. However, conventional O2O RL algorithms typically require maintaining…

D4RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Matrix Estimation for Offline Reinforcement Learning with Low-Rank Structure

2023-05-24 · Xumei Xi, Christina Lee Yu, Yudong Chen

We consider offline Reinforcement Learning (RL), where the agent does not interact with the environment and must rely on offline data collected using a behavior policy. Previous works provide policy evaluation guarantees…

Matrix Completion reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

CAWR: Corruption-Averse Advantage-Weighted Regression for Robust Policy Optimization

2025-06-18 · Ranting Hu

Offline reinforcement learning (offline RL) algorithms often require additional constraints or penalty terms to address distribution shift issues, such as adding implicit or explicit policy constraints during policy opti…

D4RL Offline RL regression