Representation Balancing Offline Model-based Reinforcement Learning

2021-01-01 · ICLR 2021 1 · Byung-Jun Lee, Jongmin Lee, Kee-Eung Kim

One of the main challenges in offline and off-policy reinforcement learning is to cope with the distribution shift that arises from the mismatch between the target policy and the data collection policy. In this paper, we focus on a model-based approach, particularly on learning the representation for a robust model of the environment under the distribution shift, which has been first studied by Representation Balancing MDP (RepBM). Although this prior work has shown promising results, there are a number of shortcomings that still hinder its applicability to practical tasks. In particular, we address the curse of horizon exhibited by RepBM, rejecting most of the pre-collected data in long-term tasks. We present a new objective for model learning motivated by recent advances in the estimation of stationary distribution corrections. This effectively overcomes the aforementioned limitation of RepBM, as well as naturally extending to continuous action spaces and stochastic policies. We also present an offline model-based policy optimization using this new objective, yielding the state-of-the-art performance in a representative set of benchmark offline RL tasks.

📄 PDF Abstract BibTeX

Code (0)

등록된 구현이 없습니다.

Tasks

modelModel-based Reinforcement LearningOffline RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Hybrid Value Estimation for Off-policy Evaluation and Offline Reinforcement Learning

2022-06-04 · Xue-Kun Jin, Xu-Hui Liu, Shengyi Jiang, Yang Yu

Value function estimation is an indispensable subroutine in reinforcement learning, which becomes more challenging in the offline setting. In this paper, we propose Hybrid Value Estimation (HVE) to reduce value estimatio…

MuJoCo Off-policy evaluation reinforcement-learning Reinforcement Learning+1

FAWAC: Feasibility Informed Advantage Weighted Regression for Persistent Safety in Offline Reinforcement Learning

2024-12-12 · Prajwal Koirala, Zhanhong Jiang, Soumik Sarkar, Cody Fleming

Safe offline reinforcement learning aims to learn policies that maximize cumulative rewards while adhering to safety constraints, using only offline data for training. A key challenge is balancing safety and performance,…

Interpretable Control by Reinforcement Learning

2020-07-20 · Daniel Hein, Steffen Limmer, Thomas A. Runkler

In this paper, three recently introduced reinforcement learning (RL) methods are used to generate human-interpretable policies for the cart-pole balancing benchmark. The novel RL methods learn human-interpretable policie…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Projected State-action Balancing Weights for Offline Reinforcement Learning

2021-09-10 · Jiayi Wang, Zhengling Qi, Raymond K. W. Wong

Offline policy evaluation (OPE) is considered a fundamental and challenging problem in reinforcement learning (RL). This paper focuses on the value estimation of a target policy based on pre-collected data generated from…

Causal Inference reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Offline Multitask Representation Learning for Reinforcement Learning

2024-03-18 · Haque Ishfaq, Thanh Nguyen-Tang, Songtao Feng, Raman Arora 외

We study offline multitask representation learning in reinforcement learning (RL), where a learner is provided with an offline dataset from different tasks that share a common representation and is asked to learn the sha…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Representation Learning