Iteratively Refined Behavior Regularization for Offline Reinforcement Learning

2023-06-09 · Xiaohan Hu, Yi Ma, Chenjun Xiao, Yan Zheng, Jianye Hao

One of the fundamental challenges for offline reinforcement learning (RL) is ensuring robustness to data distribution. Whether the data originates from a near-optimal policy or not, we anticipate that an algorithm should demonstrate its ability to learn an effective control policy that seamlessly aligns with the inherent distribution of offline data. Unfortunately, behavior regularization, a simple yet effective offline RL algorithm, tends to struggle in this regard. In this paper, we propose a new algorithm that substantially enhances behavior-regularization based on conservative policy iteration. Our key observation is that by iteratively refining the reference policy used for behavior regularization, conservative policy update guarantees gradually improvement, while also implicitly avoiding querying out-of-sample actions to prevent catastrophic learning failures. We prove that in the tabular setting this algorithm is capable of learning the optimal policy covered by the offline dataset, commonly referred to as the in-sample optimal policy. We then explore several implementation details of the algorithm when function approximations are applied. The resulting algorithm is easy to implement, requiring only a few lines of code modification to existing methods. Experimental results on the D4RL benchmark indicate that our method outperforms previous state-of-the-art baselines in most tasks, clearly demonstrate its superiority over behavior regularization.

📄 PDF Abstract BibTeX arXiv:2306.05726

Code (0)

등록된 구현이 없습니다.

Tasks

D4RL Offline RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Offline Reinforcement Learning with Fisher Divergence Critic Regularization

2021-03-14 · Ilya Kostrikov, Jonathan Tompson, Rob Fergus, Ofir Nachum

Many modern approaches to offline Reinforcement Learning (RL) utilize behavior regularization, typically augmenting a model-free actor critic algorithm with a penalty measuring divergence of the policy from the offline d…

Offline RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Offline Reinforcement Learning with Soft Behavior Regularization

2021-10-14 · Haoran Xu, Xianyuan Zhan, Jianxiong Li, Honglei Yin

Most prior approaches to offline reinforcement learning (RL) utilize \textit{behavior regularization}, typically augmenting existing off-policy actor critic algorithms with a penalty measuring divergence between the poli…

continuous-control Continuous Control reinforcement-learning Reinforcement Learning+1

Offline Reinforcement Learning with Adaptive Behavior Regularization

2022-11-15 · Yunfan Zhou, Xijun Li, Qingyu Qu

Offline reinforcement learning (RL) defines a sample-efficient learning paradigm, where a policy is learned from static and previously collected datasets without additional interaction with the environment. The major obs…

D4RL Offline RL reinforcement-learning Reinforcement Learning+1

Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularization

2023-07-21 · NeurIPS 2023 11 · Xiangsen Wang, Haoran Xu, Yinan Zheng, Xianyuan Zhan

Offline reinforcement learning (RL) has received considerable attention in recent years due to its attractive capability of learning policies from offline datasets without environmental interactions. Despite some success…

Management MuJoCo Multi-agent Reinforcement Learning Reinforcement Learning (RL)+2

B3C: A Minimalist Approach to Offline Multi-Agent Reinforcement Learning

2025-01-30 · Woojun Kim, Katia Sycara

Overestimation arising from selecting unseen actions during policy evaluation is a major challenge in offline reinforcement learning (RL). A minimalist approach in the single-agent setting -- adding behavior cloning (BC)…

Multi-agent Reinforcement Learning Reinforcement Learning (RL)