Behavior Estimation from Multi-Source Data for Offline Reinforcement Learning

2022-11-29 · Guoxi Zhang, Hisashi Kashima

Offline reinforcement learning (RL) have received rising interest due to its appealing data efficiency. The present study addresses behavior estimation, a task that lays the foundation of many offline RL algorithms. Behavior estimation aims at estimating the policy with which training data are generated. In particular, this work considers a scenario where the data are collected from multiple sources. In this case, neglecting data heterogeneity, existing approaches for behavior estimation suffers from behavior misspecification. To overcome this drawback, the present study proposes a latent variable model to infer a set of policies from data, which allows an agent to use as behavior policy the policy that best describes a particular trajectory. This model provides with a agent fine-grained characterization for multi-source data and helps it overcome behavior misspecification. This work also proposes a learning algorithm for this model and illustrates its practical usage via extending an existing offline RL algorithm. Lastly, with extensive evaluation this work confirms the existence of behavior misspecification and the efficacy of the proposed model.

📄 PDF Abstract BibTeX arXiv:2211.16078

Code (1)

Altriaex/multi_source_behavior_modeling 공식 구현 tf

Tasks

Offline RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Peng's Q($λ$) for Conservative Value Estimation in Offline Reinforcement Learning

2026-05-14 · Byeongchan Kim, Min-hwan Oh arxiv

We propose a model-free offline multi-step reinforcement learning (RL) algorithm, Conservative Peng's Q($λ$) (CPQL). Our algorithm adapts the Peng's Q($λ$) (PQL) operator for conservative value estimation as an alternati…

Reinforcement Learning Offline RL

OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimation

2021-06-21 · Jongmin Lee, Wonseok Jeon, Byung-Jun Lee, Joelle Pineau 외

We consider the offline reinforcement learning (RL) setting where the agent aims to optimize the policy solely from the data without further environment interactions. In offline RL, the distributional shift becomes the p…

Offline RL Reinforcement Learning (RL)

Behavior Proximal Policy Optimization

2023-02-22 · Zifeng Zhuang, Kun Lei, Jinxin Liu, Donglin Wang 외

Offline reinforcement learning (RL) is a challenging setting where existing off-policy actor-critic methods perform poorly due to the overestimation of out-of-distribution state-action pairs. Thus, various additional aug…

D4RL Offline RL Reinforcement Learning (RL)

BRAC+: Going Deeper with Behavior Regularized Offline Reinforcement Learning

2021-01-01 · Chi Zhang, Sanmukh Rao Kuppannagari, Viktor Prasanna

Online interactions with the environment to collect data samples for training a Reinforcement Learning agent is not always feasible due to economic and safety concerns. The goal of Offline Reinforcement Learning (RL) is …

Offline RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

B3C: A Minimalist Approach to Offline Multi-Agent Reinforcement Learning

2025-01-30 · Woojun Kim, Katia Sycara

Overestimation arising from selecting unseen actions during policy evaluation is a major challenge in offline reinforcement learning (RL). A minimalist approach in the single-agent setting -- adding behavior cloning (BC)…

Multi-agent Reinforcement Learning Reinforcement Learning (RL)