Behavior-Adaptive Q-Learning: A Unifying Framework for Offline-to-Online RL

2025-11-05 · Lipeng Zu, Hansong Zhou, Xiaonan Zhang arxiv

Offline reinforcement learning (RL) enables training from fixed data without online interaction, but policies learned offline often struggle when deployed in dynamic environments due to distributional shift and unreliable value estimates on unseen state-action pairs. We introduce Behavior-Adaptive Q-Learning (BAQ), a framework designed to enable a smooth and reliable transition from offline to online RL. The key idea is to leverage an implicit behavioral model derived from offline data to provide a behavior-consistency signal during online fine-tuning. BAQ incorporates a dual-objective loss that (i) aligns the online policy toward the offline behavior when uncertainty is high, and (ii) gradually relaxes this constraint as more confident online experience is accumulated. This adaptive mechanism reduces error propagation from out-of-distribution estimates, stabilizes early online updates, and accelerates adaptation to new scenarios. Across standard benchmarks, BAQ consistently outperforms prior offline-to-online RL approaches, achieving faster recovery, improved robustness, and higher overall performance. Our results demonstrate that implicit behavior adaptation is a principled and practical solution for reliable real-world policy deployment.

📄 PDF Abstract BibTeX arXiv:2511.03695

Code (0)

등록된 구현이 없습니다.

Tasks

Reinforcement Learning

Cognitive Duality for Adaptive Web Agents

2025-08-07 · Jiarun Liu, Chunhong Zhang, Zheng Hu arxiv

Web navigation represents a critical and challenging domain for evaluating artificial general intelligence (AGI), demanding complex decision-making within high-entropy, dynamic environments with combinatorially explosive…

Adaptive Behavior Cloning Regularization for Stable Offline-to-Online Reinforcement Learning

2022-10-25 · Yi Zhao, Rinu Boney, Alexander Ilin, Juho Kannala 외

Offline reinforcement learning, by learning from a fixed dataset, makes it possible to learn agent behaviors without interacting with the environment. However, depending on the quality of the offline dataset, such pre-tr…

D4RL Offline RL reinforcement-learning Reinforcement Learning+1

Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimization

2023-11-06 · Kun Lei, Zhengmao He, Chenhao Lu, Kaizhe Hu 외

Combining offline and online reinforcement learning (RL) is crucial for efficient and safe learning. However, previous approaches treat offline and online learning as separate procedures, resulting in redundant designs a…

Deep Reinforcement Learning Reinforcement Learning (RL)

Policy Expansion for Bridging Offline-to-Online Reinforcement Learning

2023-02-02 · Haichao Zhang, We Xu, Haonan Yu

Pre-training with offline data and online fine-tuning using reinforcement learning is a promising strategy for learning control policies by leveraging the best of both worlds in terms of sample efficiency and performance…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

COOPO: Cyclic Offline-Online Policy Optimization Algorithm

2026-05-18 · Qisai Liu, Zhanhong Jiang, Joshua Russell Waite, Aditya Balu 외 arxiv

Offline reinforcement learning struggles with distributional shift and constrained performance due to static dataset limitations, while online RL demands prohibitive environment interactions. The recent advent of hybrid …