A Behavior Regularized Implicit Policy for Offline Reinforcement Learning

2022-02-19 · Shentao Yang, Zhendong Wang, Huangjie Zheng, Yihao Feng, Mingyuan Zhou

Offline reinforcement learning enables learning from a fixed dataset, without further interactions with the environment. The lack of environmental interactions makes the policy training vulnerable to state-action pairs far from the training dataset and prone to missing rewarding actions. For training more effective agents, we propose a framework that supports learning a flexible yet well-regularized fully-implicit policy. We further propose a simple modification to the classical policy-matching methods for regularizing with respect to the dual form of the Jensen--Shannon divergence and the integral probability metrics. We theoretically show the correctness of the policy-matching approach, and the correctness and a good finite-sample property of our modification. An effective instantiation of our framework through the GAN structure is provided, together with techniques to explicitly smooth the state-action mapping for robust generalization beyond the static dataset. Extensive experiments and ablation study on the D4RL benchmark validate our framework and the effectiveness of our algorithmic designs.

📄 PDF Abstract BibTeX arXiv:2202.09673

Code (0)

등록된 구현이 없습니다.

Tasks

D4RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

State-Action Joint Regularized Implicit Policy for Offline Reinforcement Learning

2021-09-29 · Shentao Yang, Zhendong Wang, Huangjie Zheng, Mingyuan Zhou

D4RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

BRAC+: Improved Behavior Regularized Actor Critic for Offline Reinforcement Learning

2021-10-02 · Chi Zhang, Sanmukh Rao Kuppannagari, Viktor K Prasanna

Online interactions with the environment to collect data samples for training a Reinforcement Learning (RL) agent is not always feasible due to economic and safety concerns. The goal of Offline Reinforcement Learning is …

Offline RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Reinforcement Learning via Value Gradient Flow

2026-04-15 · Haoran Xu, Kaiwen Hu, Somayeh Sojoudi, Amy Zhang arxiv

We study behavior-regularized reinforcement learning (RL), where regularization toward a reference distribution (the dataset in offline RL or the base model in LLM RL finetuning) is essential to prevent value over-optimi…

Reinforcement Learning Offline RL

Model-Based Offline Meta-Reinforcement Learning with Regularization

2022-02-07 · ICLR 2022 4 · Sen Lin, Jialin Wan, Tengyu Xu, Yingbin Liang 외

Existing offline reinforcement learning (RL) methods face a few major challenges, particularly the distributional shift between the learned policy and the behavior policy. Offline Meta-RL is emerging as a promising appro…

Meta Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1

Behavior Constraining in Weight Space for Offline Reinforcement Learning

2021-07-12 · Phillip Swazinna, Steffen Udluft, Daniel Hein, Thomas Runkler

In offline reinforcement learning, a policy needs to be learned from a single pre-collected dataset. Typically, policies are thus regularized during training to behave similarly to the data generating policy, by adding a…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)