Adaptive Behavior Cloning Regularization for Stable Offline-to-Online Reinforcement Learning

2022-10-25 · Yi Zhao, Rinu Boney, Alexander Ilin, Juho Kannala, Joni Pajarinen

Offline reinforcement learning, by learning from a fixed dataset, makes it possible to learn agent behaviors without interacting with the environment. However, depending on the quality of the offline dataset, such pre-trained agents may have limited performance and would further need to be fine-tuned online by interacting with the environment. During online fine-tuning, the performance of the pre-trained agent may collapse quickly due to the sudden distribution shift from offline to online data. While constraints enforced by offline RL methods such as a behaviour cloning loss prevent this to an extent, these constraints also significantly slow down online fine-tuning by forcing the agent to stay close to the behavior policy. We propose to adaptively weigh the behavior cloning loss during online fine-tuning based on the agent's performance and training stability. Moreover, we use a randomized ensemble of Q functions to further increase the sample efficiency of online fine-tuning by performing a large number of learning updates. Experiments show that the proposed method yields state-of-the-art offline-to-online reinforcement learning performance on the popular D4RL benchmark. Code is available: \url{https://github.com/zhaoyi11/adaptive_bc}.

📄 PDF Abstract BibTeX arXiv:2210.13846

Code (2)

zhaoyi11/adaptive_bc 공식 구현 pytorch

zzmtsvv/rl_task pytorch

Tasks

D4RL Offline RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Offline Reinforcement Learning with Adaptive Behavior Regularization

2022-11-15 · Yunfan Zhou, Xijun Li, Qingyu Qu

Offline reinforcement learning (RL) defines a sample-efficient learning paradigm, where a policy is learned from static and previously collected datasets without additional interaction with the environment. The major obs…

D4RL Offline RL reinforcement-learning Reinforcement Learning+1

Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RL

2025-05-26 · Qin-Wen Luo, Ming-Kun Xie, Ye-Wen Wang, Sheng-Jun Huang

Offline reinforcement learning (RL) aims to learn an effective policy from a static dataset. To alleviate extrapolation errors, existing studies often uniformly regularize the value function or policy updates across all …

D4RL Offline RL Reinforcement Learning (RL)

Know Your Boundaries: The Necessity of Explicit Behavioral Cloning in Offline RL

2022-06-01 · Wonjoon Goo, Scott Niekum

We introduce an offline reinforcement learning (RL) algorithm that explicitly clones a behavior policy to constrain value learning. In offline RL, it is often important to prevent a policy from selecting unobserved actio…

D4RL Offline RL Reinforcement Learning (RL)

B3C: A Minimalist Approach to Offline Multi-Agent Reinforcement Learning

2025-01-30 · Woojun Kim, Katia Sycara

Overestimation arising from selecting unseen actions during policy evaluation is a major challenge in offline reinforcement learning (RL). A minimalist approach in the single-agent setting -- adding behavior cloning (BC)…

Multi-agent Reinforcement Learning Reinforcement Learning (RL)

A Connection between One-Step Regularization and Critic Regularization in Reinforcement Learning

2023-07-24 · Benjamin Eysenbach, Matthieu Geist, Sergey Levine, Ruslan Salakhutdinov

As with any machine learning problem with limited data, effective offline RL algorithms require careful regularization to avoid overfitting. One-step methods perform regularization by doing just a single step of policy i…

Offline RL reinforcement-learning