The Three Regimes of Offline-to-Online Reinforcement Learning

2025-10-01 · Lu Li, Tianwei Ni, Yihao Sun, Pierre-Luc Bacon arxiv

Offline-to-online reinforcement learning (RL) has emerged as a practical paradigm that leverages offline datasets for pretraining and online interactions for fine-tuning. However, its empirical behavior is highly inconsistent: design choices of online fine-tuning that work well in one setting can fail completely in another. Guided by the stability--plasticity principle, we propose a framework that can explain this inconsistency: We argue that efficient fine-tuning must preserve the utility of the stronger offline prior, whether that is the pretrained policy or the offline dataset, while maintaining sufficient plasticity. This perspective identifies three regimes of online fine-tuning, each requiring distinct stability properties. We validate this framework through a large-scale empirical study, finding that the results strongly align with its predictions in 45 out of 63 cases, with only 3 opposite mismatches. This work provides a framework for guiding design choices in offline-to-online RL based on the relative performance of the offline dataset and the pretrained policy.

📄 PDF Abstract BibTeX arXiv:2510.01460

Code (0)

등록된 구현이 없습니다.

Tasks

Reinforcement Learning

Addressing Extrapolation Error in Deep Offline Reinforcement Learning

2021-01-01 · Caglar Gulcehre, Sergio Gómez Colmenarejo, Ziyu Wang, Jakub Sygnowski 외

Reinforcement learning (RL) encompasses both online and offline regimes. Unlike its online counterpart, offline RL agents are trained using logged-data only, without interaction with the environment. Therefore, offline…

Offline RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Bridging Offline and Online Reinforcement Learning for LLMs

2025-06-26 · Jack Lanchantin, Angelica Chen, Janice Lan, Xian Li 외

We investigate the effectiveness of reinforcement learning methods for finetuning large language models when transitioning from offline to semi-online to fully online regimes for both verifiable and non-verifiable tasks.…

Instruction Following Math reinforcement-learning Reinforcement Learning

On the Role of Discount Factor in Offline Reinforcement Learning

2022-06-07 · Hao Hu, Yiqin Yang, Qianchuan Zhao, Chongjie Zhang

Offline reinforcement learning (RL) enables effective learning from previously collected data without exploration, which shows great promise in real-world applications when exploration is expensive or even infeasible. Th…

D4RL Offline RL reinforcement-learning Reinforcement Learning+1

Leveraging Offline Data in Online Reinforcement Learning

2022-11-09 · Andrew Wagenmaker, Aldo Pacchiano

Two central paradigms have emerged in the reinforcement learning (RL) community: online RL and offline RL. In the online RL setting, the agent has no prior knowledge of the environment, and must interact with it in order…

Offline RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Competitive Perimeter Defense in Tree Environments

2024-07-24 · Richard L. Frost, Shaunak D. Bopardikar

We consider a perimeter defense problem in a rooted full tree graph environment in which a single defending vehicle seeks to defend a set of specified vertices, termed as the perimeter from mobile intruders that enter th…